Experiments

class: center, middle, inverse, title-slide

# Experiments
## You’re in Control!
### Updated 2020-08-21

---

# Random experiments and causality

- Annoyingly, most things we'd like to know the effect of have back-door paths to most of the outcomes we'd like to know the effect *on*

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAIAAADOgk3lAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dXWwTZ7oH8LEX22RRJBII0EVbUGbsJC3JgQI9gl6woT0bO5xFvajYUhzosoF820D2XKyqI9QLTq9AkEBoQtru0TahBCpVqIc4cT5gJYpEFWjsYAePJ6I9AkMczJEQJOPQ+Fy8WzfNp+3Yfj0z/98VTSfjf2yPHz8z7zuvKhgMMgAAAFKjph0AAAAgFihgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgAAAgSShgEIVly5Zptdpdu3bRDhKRXbt2JSJtb2+vdj7Hjh1b4KMcO3ZMq9UWFBTEJTNd01+IXbt2yeNPA7pQwCBSvb29T58+ZRimu7ubdhaQsGPHjn311Ve0U4AcLKIdACTDarUyDJObmzs4ONjb21tYWEg70Tza2toSt/P09PTHjx8nbv9yktAXApQMHRhEanBwMD09/Y9//CPzUzEDAKAIBQwiQi7qvPnmmx988AHDMIODg9O3KSgoIJd/jh07Rq6Wkas4vb29UW1D9Pb2ko1n3IZcIlq2bNmU3yKXW8j1lSmXXiJ/6PDjLlu2rLe3dyHX0iJ/UPIXzX0Jbe7nJKr8U3ZFtowt+by7mhxg2bJlH374IcMwg4ODZIeRvJQAMwsCRCA3N5dhmI6OjvC/jx49OuM26enpc7zNItkmGAwePXp0xrfr5Aed8Q1M9kw2e/vttxmGefvtt6N66OkbkF8M76ejo4PsJ/Lnbd4HJZtNf9Dc3NyonpNI8of/hLl3FWHySHY1+YWYskPyB07f7ZSXEmBG6MBgfr29veT8IbnuRc4injhxYsaNnz59Gv7YDX/mTmkp5t6mt7eXfEnPzc0lJTP40+fphx9+GP52T34yfc8Mw5A2MYZ4u3btInsgn5sdHR3p6ekztptRmftBjx07Rh4i/GFNLjRO3kOEz0mE+YuKiphfVkdSY8hDRJ48ql0Rjx8/Jvshv+JwOJhYX0oAFDCY39mzZxmGefPNN8l/ks+Up0+fznj+Kj09nXwqkS3JZ1N/f3/k25CHy83NdTgc4aEiDoeDbBa+/Ebq6IULF8K7JZ+A07uZyB+aDLA8evQo+RsLCwsvXbo0436ePn062xj66WfD5n5Q8ie8/fbb4Q9rh8MxpVOJ8DmJJD85lUd2Ff5hW1sbKTxTzjTOnTyqXc0htpcSAAUM5kc+FiePJSOfLORTdYrVq1dP/k+y5ZQmYO5tyMORD7XJyE/u379P/nP61TjyCTj9FyN8aDJPID09ffK3/sLCwoV/jM7995J/VFRUTN4m/HWBiOQ5iTA/2dWpU6em7IoEmDJHIpJXKsJdzSG2lxIABQzmQT4Wp3wIkk+WGWfzRPJxP/c25NzR1q1bp/w83PlN2U/41BP5BJz7pNMcD/3NN98w0z6yZ/uVOa6BTR9eH8lzMmVawr/8y79M/s9InpOo8hcVFU1pHMnJwMlPb4TJI9zV3GJ4KQFQwGAepM0iY8bCwlc4Fn7LidlEMs9s8qkniZ50mvE07GziMvcuqrqStF3J4KWE5EMBg3nMfdOEydct4iuST3byDZ2cQIvXSafwKcrkiKomRfKcRJg/PBJkusjzxHFXiXgpQfZQwGAu4e/C0z+byPhpcleO+D4oGb9ATohNDzN9HDYZTkI++xZy0mn6CTpi4aMQIzHlaZw+7IWZ7zmJMP9su4pBHHfFxPWlBIVAAYO5zPFduLCwkHx+zTiUYyHI+IXpvR35yZRrPOE7g0y/UBct8hc9ffp0ynnRRBewGUfETBkBEclzEmF+sqvpsyBmm1A8hzjuionrSwkKgQIGc5n7Wjr5/Ir7vX3JGLbBwcHwfR/IvR5ImClj3iYPYFv4px75i8Izq3p7e6P9FI4B+Yu++uqryTcNmXJ5KcLnJJL8ZDTp06dPJ98y49ixY+S65pEjRyJPvpBdTT/VGd+XEpQABQxmFZ7lM9sG5FN1+lf+BSosLCRzXQcHB8kIt6KiovBU3+kXjcInFaeMRI9BW1sb2Vv4cWfrBuaYBxZt81FYWEgmTn311Vfk1wcHB6c8aITPSYT5ya6ePn0aHkAYniUd7Ym7GHZFTnWSJ3DKExXHlxKUAAUMZjXb3KOw8FnEuA/l+OCDDzo6OiZ/8pI7UMz4mUjajvCNQhbo8ePH4cdNT0+f7VZJ8dXW1nb06NHwx/fRo0enP+0RPieR5J++q/T09KNHj06ejxyhGHYVLtjTxfelBNlTxTDoCCCl7Nq166uvvnr77bcTtGwHOVMXvr2F5Egof6JfSpAZdGAgeaRTXPhJp/BSy1N+Ts7UzTbGL3VIPT8Tv5cSFAIFDKTt2LFj5DLPwk86hfcweQkP8m9JnNSSev44vpSgEFiRGaQqPAaPmel2fLE5evTohx9+SG47Mvnns93SN9VINH8iXkpQAnRgIFXhgQMzDk2MzQcffDB5MAXz00CJefcfDAYjnC42ODiYuAvPMeenKxEvJSgBBnEAxMFnn31WXl7+ySefmM3mOTb7+9//Xlpa2tjY+P777ycrGoBsoYABLNTExER+fr7P5xMEISMjY44tA4EAy7K/+c1vnE6nWo3zHwALgkMIYKE6Ojp4ni8pKZm7ejEMk5mZWVJSwvN8Z2dncrIByBgKGMBCNTQ0qNXqqqqqSDauqqpSq9UNDQ2JTgUgeyhgAAvidrs7OztNJpPBYIhk+5ycHJPJ1NHRkZyb3APIGAoYwIKcPHkyFApFdQ/cw4cPh0KhkydPJi4VgBJgEAdA7B49esRxXH5+frRrYm3dutXpdHq93pUrVyYoG4DsoQMDiN1nn30mimJ5eXm0v1hWViaK4t/+9rcEhAJQCnRgADEKBoN6vX5iYsLr9ep0uqh+VxRFjuPUajXP89PvXggAkUAHBhCjlpYWn89XVVUVbfViGEan01VWVvp8vtbW1kRkA1ACdGAAsQiFQhs2bPj++++Hhobmnf41IzKpee3atbdu3VKpVHFPCCB76MAAYmG3210ul9lsjq16MQyTmZm5Z8+eO3fudHV1xTcbgEKggAHE4tSpU2q12mq1LmQnVqtVrVbj/usAsUEBA4iaw+Gw2+0mk4njuIXsR6/XG43Gzs5Op9MZr2wAyoECBhC1+vp6hmEsFsvCd0V2QnYIAFHBIA6A6Ph8PoPBkJub++2338Zlh5s3bx4cHOR5ftWqVXHZIYBCoAMDiE5TU5MoinFpv4iamhpRFJuamuK1QwCFQAcGEIVnz56xLKvT6eI4AZlMiH7x4oXX601LS4vLPgGUAB0YQBTa2toCgUBpaWkcb5+h1WpLS0v9fv+FCxfitU8AJUAHBhCpUCj02muvCYLg9XpXrFgRxz0PDw9zHMdxXF9fHyY1A0QIHRhApDo7O+/cubNv3774Vi+GYVasWLF3796BgQG73R7fPQPIGAoYQKTq6+sXPnl5NmRSc11dXSJ2DiBLKGAAEbl7967dbi8qKlrg5OXZ6PX63//+911dXR6PJxH7B5AfFDCAiJw4cSLalZejdeTIkYmJiRMnTiTuIQDkBIM4AOb38OFDvV4fw8rL0Xr99dfdbjcmNQNEAh0YwPzI5OXq6upEP5DVahVF8dy5c4l+IAAZQAcGMI/R0VGO47Rarcfj0Wg0CX2s8fFxg8EQDAYxqRlgXujAAObR0tLi9/vLy8sTXb0YhtFoNGVlZX6/Hys1A8wLHRjAXCYmJgoKCu7fvx/zysvRIis1r1692uFwqNX4igkwKxweAHOx2Wwej6ekpCQ51YthmMzMTLPZ7PF4Ojo6kvOIABKFDgxgLkaj8erVqwMDAwma/jUjr9e7bt26wsLC9vb2pD0ogOSgAwOYVX9/f09Pz8JXXo4Wx3Emk6m7u7u/vz+ZjwsgLShgALM6c+YMwzBVVVXJf+jKykqGYRoaGpL/0ABSgVOIADN79OgRx3F6vZ7KHeITd+d7ANlABwYws7Nnz4qiWFtbS2V9E5VKVVtbOzY2dvbs2eQ/OoAkoAMDmMHo6CjLsosXL757924Spn/NCJOaAeaGDgxgBm1tbSMjIwcOHKBVvRiG0Wg0Bw4c8Pv9Fy9epJUBIJWhAwOYKhQKbdq0SRAEQRCWLVtGMcnIyAjHcSzL9vX1UYwBkJrQgQFM1dHR4XQ6//SnP9GtXgzDLF++/P3333c6nZjUDDAdOjCAqYqLi3t7e91u99q1a2lnYe7du5eXl7d9+/b/+Z//oZ0FILWgAwP4BYfD0dXVtWPHjlSoXgzDrF27tri42G63O51O2lkAUgsKGMAvnD59mmEYq9VKO8jPSBgSDADCcAoR4Gdk5eWCgoLr16/TzvILb7zxhsPhwErNAJOhAwP4WXNzsyiK5DZOKaWiokIUxU8++YR2EIAUgg4M4J9EUeQ4Tq1W8zyv1Wppx/mFYDCo1+tDoRDP8zqdjnYcgJSADgzgn1paWh49elRTU5Nq1YthGK1WW11d/fDhQ6zUDBCGDgyAYRgmFApt2LDhhx9+GBoaWrp0Ke04M3jy5AnLsi+//PLt27ep3J4RINWgAwNgGIa5evWqy+XavXt3alYvhmEyMjLeffddl8t17do12lkAUgIKGADDMMzJkydVKlVNTQ3tIHOpqalRqVQnT56kHQQgJaCAATAOh6O9vb24uDg3N5d2lrnk5eWZTKYrV65gUjMAgwIGwDBMfX09wzAp3n4RJCQJDKBwGMQBSuf3+1mWzc7OlsTgiFAotH79+nv37nm93qysLNpxAGhCBwZK19DQMDY2dvjw4dSvXgzDqFSqw4cPj46OYqVmAHRgoGjPnj1jWVan03k8HqlMEBZFUa/X//jjj1ipGRQOHRgoWmtrayAQOHjwoFSqF8MwOp3u4MGDfr+/paWFdhYAmtCBgXJNTEzk5+c/ePBAEITMzEzacaLg9/s5jvvtb3/rcDjUanwNBYXCWx+Uy2az8Tz/3nvvSat6MQyTlZW1Z88ej8eDlZpBydCBgXIZjcarV686nU69Xk87S9Q8Hk9BQUFhYWF7ezvtLAB0oAMDhXK5XL29vW+99ZYUqxfDMAaD4c033+zp6XG73bSzANCBAgYKVVdXFwqFUmrl5WhZrdZQKFRXV0c7CAAdOIUISkRWXs7Ly7t58ybtLAuyefPmwcFBrNQMyoQODJTo3LlzoihKuv0irFarKIrnzp2jHQSAAnRgoDiiKBoMBnmsbkwmNatUqhRcRRog0dCBgeK0tLT4fL6qqiqpVy+GYXQ6XWVlpc/nw6RmUCB0YKAs4ZWXBUHIyMigHScOnjx5kp2dvWbNGkncjBggjtCBgbJ0dna6XC6z2SyP6sUwTEZGhtlsdrlcdruddhaApEIBA2U5deqUWq22WCy0g8ST1WpVq9WnTp2iHQQgqVDAQEEcDkdXV5fJZOI4jnaWeOI4zmQy2e12h8NBOwtA8qCAgYKQOb8yGD0/HekpsVIzKAoGcYBS+Hw+vV7/yiuvSH3y8mxef/11t9vt8Xheeukl2lkAkgEdGChFY2NjMBiUZftFWCwWURSbmppoBwFIEnRgoAhk5eW0tDSPx6PRaGjHSYhgMKjX60VRFARhyZIltOMAJBw6MFCECxcuBAKBP//5z3KtXgzDaLXa0tLSQCDQ1tZGOwtAMqADA/kLhULr16+/d++e1+vNysqiHSeBhoeHOY7Lzs7GpGZQAnRgIH82m83tdu/du1fe1YthmBUrVuzdu9flcmGlZlACFDCQv/r6erVaLePhG5MdOnRIrVZjPD0oAQoYyNzg4GB3d7fRaJTZ5OXZcBxXVFTU1dV19+5d2lkAEgsFDGROBisvRwsrNYNCYBAHyBlZebmgoOD69eu0syTVG2+84XA4sFIzyBs6MJCzpqYmURRrampoB0m26upqrNQMsocODGRrdHSU4zidTufxeBYtWkQ7TlKNj4/n5OSIouj1etPS0mjHAUgIdGAgW62trX6/v6KiQmnVi2EYjUZTXl7u9/vPnz9POwtAoqADA3mamJgoKCh48ODB0NDQ0qVLaceh4MmTJyzLvvzyy5jUDHKFDgzkqaenx+Px7NmzR5nVi2GYjIyM9957z+Vy9fT00M4CkBAoYCBPdXV18lt5OVoWi0WtVmM8PcgVChjIkNPptNls//7v/67X62lnoclgMOzYsaO9vd3pdNLOAhB/KGAgQ6dPn2ZkuvJytMiTcObMGdpBAOIPgzhAbh4/fsyyLMuyfX19tLOkhI0bNwqCIAjCsmXLaGcBiCd0YCA3jY2Nz58/R/sVZrFYnj9/jpWaQX7QgYGskMnLWq1WxisvR4us1Pzjjz/yPI9JzSAn6MBAVsjk5QMHDqB6hWm12oMHDw4PD2NSM8gMOjCQj1AotHHjRp7nvV7vypUracdJIY8ePeI4Tq/X9/X1YVIzyAY6MJCPjo6OgYGBkpISVK8pVq5caTabBwYGOjs7aWcBiBsUMJAPMnn58OHDtIOkosOHD2NSM8gMChjIhMPh6OrqMhqNCp+8PBuDwVBUVGS32zGpGWQDBQxkApOX50WeHPJEAcgABnGAHJCVl/Py8m7evEk7S0rbvHnz4OAgVmoGeUAHBnJAVl5W+K17I2GxWLBSM8gGOjCQPExejhyZ1PzixQus1AwygA4MJO/8+fN+v7+8vBzVa15arbaiosLv93/xxRe0swAsFDowkLZQKLRhw4YffvhBEISMjAzacSTgyZMn2dnZa9aswUrNIHXowEDa7Ha7y+UqKSlB9YpQRkZGSUmJy+Wy2+20swAsCAoYSNuZM2dUKlVlZSXtIFJSWVmpUqkaGhpoBwFYEBQwkDCHw9He3r5jx46cnBzaWaQkNze3uLj4ypUrDoeDdhaA2KGAgYSRGyNh9HwMyJNWX19POwhA7DCIA6RqeHiY4ziWZW/duoXBCNEig1+Ghoa8Xu+KFStoxwGIBTowkKrGxsaxsTGLxYLqFQOVSmWxWMbGxhobG2lnAYgROjCQpGfPnrEsq9PpeJ7XarW040gSmdQ8Pj4uCAImNYMUoQMDSWptbQ0EAuXl5aheMdNqtWVlZSMjIy0tLbSzAMQCHRhIz8TERH5+/oMHDwRByMzMpB1HwgKBAMuyq1evdjgcajW+zoLE4C0L0tPR0cHzfElJCarXAmVmZprNZo/Hg5WaQYrQgYH0mEym3t7egYEBjuNoZ5E8r9e7bt267du3X7lyhXYWgOigAwOJcTgc3d3dxcXFqF5xwXGcyWTq6urCpGaQHBQwkBgy9/bQoUO0g8gHeTKxUjNIDk4hgpSQlZcLCgquX79OO4usbN261el0YqVmkBZ0YCAlZOXl6upq2kHkprq6Gis1g+SgAwPJEEWR4zi1Wo3Jy3FHJjWHQiGe53U6He04ABFBBwaScf78+UePHlVUVKB6xZ1Wqy0vL3/48CFWagYJQQcG0kAmL/t8vqGhoaVLl9KOI0NkpWZMagYJwdsUpMFms/E8bzabUb0SJCMjg0xq7ujooJ0FICIoYCANZOXliooK2kHkrKKiQqVSnTlzhnYQgIiggIEE9Pf32+324uLivLw82lnk7JVXXikuLu7s7MSkZpAEFDCQADJ5+fDhw7SDyB+Z1IyVmkESMIgDUp3f72dZNicn59tvv6WdRRE2bdrE87zX683KyqKdBWAu6MAg1TU1NY2NjeHeUUlz6NCh0dFRTGqG1IcODFLa6Ogox3E6nc7j8SxatIh2HEUYHx/PyckRRVEQhMWLF9OOAzArdGCQ0r744gu/319RUYHqlTQajaa8vNzv92NSM6Q4dGCQ0jZv3uzxeARBWL58Oe0sCjIyMsKyrMFgwHVHSGXowCB19fT09Pf379mzB9UryZYvX/7ee+/19/f39vbSzgIwK3RgkLp27NjR3d2NlZep4Hk+Pz//rbfe+vrrr2lnAZgZOjBIUQ6Hw263G41GVC8q9Hp9UVERJjVDKkMBgxR19uxZhmFw7yiKyJP/8ccf0w4CMDOcQoRURFZezs3NxSACujZt2nT37l2s1AypCR0YpCKy8nJNTQ3tIEpXU1ODlZohZaEDg5RDJi8vWrQIKy9TJ4qiwWB48eKF1+tNS0ujHQfgF9CBQcppbW31+/3l5eWoXtTpdLqysjK/33/+/HnaWQCmQgcGqWViYqKgoOD+/fuCIGRmZtKOA0wgEGBZds2aNbdv31apVLTjAPwMHRikFrvd7vF4zGYzqleKyMzM3LNnj8vlstvttLMA/AIKGKSWuro6tVpttVppB4GfWa1WtVpdV1dHOwjAL6CAQQpxOp12u91kMmHyckrR6/Umk6mzs9PpdNLOAvAzFDBIIadPn2Z+WhQYUgrpic+cOUM7CMDPMIgDUgWZvJyfn//NN9/QzgIzeP31191uNyY1Q+pABwap4pNPPhFFEfeOSlnV1dWiKH766ae0gwD8EzowSAlk8rJWq/V4PBqNhnYcmMH4+LjBYAgGg5jUDCkCHRikhJaWFr/fX1ZWhuqVsjQazcGDB/1+f2trK+0sAAyDDgxSQSgUeu211wRBEAQhKyuLdhyY1fDwMMdxLMveunULk5qBOnRgQJ/dbr9z587u3btRvVLcihUr3n333Tt37nR1ddHOAoAODFKA0Wi8evWq0+nU6/W0s8A8yErNhYWF7e3ttLOA0qEDA8r6+/t7enpMJhOqlyTo9Xqj0djd3Y2VmoE6FDCgrL6+nvlpnixIAnmxyAsHQBFOIQJNPp/PYDDk5eXdvHmTdhaIAiY1QypABwY0kZWX0X5JjsViEUWxqamJdhBQNHRgQM3o6CjLsjqdDpOXJQeTmiEVoAMDalpaWkZGRioqKlC9JEej0ZSXl2NSM9CFDgzoICsvP3jwQBCEjIwM2nEgak+ePGFZ9re//e13332HSc1ABTowoKOzs9Pj8ezduxfVS6IyMjJKSkrcbndnZyftLKBQKGBAR0NDg1qtrqyspB0EYldZWalSqRoaGmgHAYVCAQMKXC5XR0eHyWQyGAy0s0DscnJyTCaTzWZzu920s4ASoYABBSdOnAiFQocPH6YdBBbq8OHDoVDoxIkTtIOAEmEQBySbz+fT6/UFBQVYeVkeyKRmj8fz0ksv0c4CyoIODJLt3LlzwWCwrKyMdhCIj8rKSlEUm5ubaQcBxUEHBkn17NkzlmXT0tIweVk2yKRmURQFQcCkZkgmdGCQVOfPnw8EApi8LCdkUvPIyAgmNUOSoQOD5AmFQhs3bhwaGvJ6vcuXL6cdB+JmZGSE47js7Oy+vj5MaoakQQcGydPT0zMwMFBSUoLqJTPLly83m80DAwO9vb20s4CCoAOD5NmxY0dPT4/b7V67di3tLBBngiC8+uqrb7311tdff007CygFOjBIEqfTabfbi4uLUb1kiWXZ4uLizs5Op9NJOwsoBQoYJEljYyPDMFVVVbSDQKKQFxeLhEHS4BQiJAMu8isBBulAkqEDg2RoaGh4/vz5kSNHUL1kTKVSHT58+Pnz52fPnqWdBRQBHRgk3OjoKMdxixYt4nleq9XSjgMJFAwG9Xr9ixcvsFIzJAE6MEi4Cxcu+P3+0tJSVC/Z02q1f/7zn/1+f1tbG+0sIH/owCCxQqFQfn7+/fv3BUHIzMykHQcSzu/3cxy3du1arNQMiYYODBLLbrd7PJ49e/ageilEVlaW2Wx2u91dXV20s4DMoYBBYjU0NKhUKoyeV5Sqqiqs1AxJgAIGCXT37t329naj0ZiXl0c7CyTPK6+8UlRU1N7e7vF4aGcBOUMBgwQiKy8fOXKEdhBItiNHjkxMTGClZkgoDOKARHn48KFer8/Ly7t58ybtLEABWamZ5/lVq1bRzgLyhA4MEuXcuXOiKFqtVtpBgA6LxSKK4rlz52gHAdlCBwYJQSYva7VarLysWJjUDImGDgwSorW11e/3l5WVoXopllarLS8v9/v958+fp50F5AkdGMTfxMREQUHB/fv3h4aGMjIyaMcBagKBAMuyq1evdjgcajW+LkOc4S0F8dfR0eHxeMxmM6qXwmVmZu7Zs8fj8XR2dtLOAjKEDgziz2g0Xr16dWBggOM42lmAMp7n8/PzCwsL29vbaWcBuUEHBnHW39/f09NjMplQvYBhGL1ebzQau7u7+/v7aWcBuUEBgzg7c+YMwzCVlZW0g0CqIG8G3FkK4g6nECGehoeHOY5jWfbWrVu4EzkQoVDotddeEwRBEISsrCzacUA+0IFBPJ09e3ZsbOwvf/kLqheEqVSq2trasbExrNQM8YUODOIGk5dhNuPj4waDIRgMCoKwePFi2nFAJtCBQdxcvHjR7/cfOHAA1Qum0Gg0paWlfr//4sWLtLOAfKADg/ggk5cfPHggCAKmf8F0T548YVn25Zdfvn37Ns4wQ1ygA4P46Ozs9Hg8JSUlqF4wo4yMDLPZ7HK5MKkZ4gUFDOKjrq5OrVbX1NTQDgKpy2KxqNXquro62kFAJlDAIA4cDkdXV1dxcTEmL8McOI4zmUx2u93hcNDOAnKAAgZxQL5TY+kvmBd5k9TX19MOAnKAQRywUD6fT6/XFxQUfPPNN7SzgASQlZo9Hs9LL71EOwtIGzowWKjGxsZgMFhdXU07CEiD1WoVRbGpqYl2EJA8dGCwIM+ePWNZNi0tDZOXIUJkUvPo6KggCEuWLKEdByQMHRgsyOeffx4IBMrLy1G9IEIajaasrCwQCLS0tNDOAtKGDgxiNzExsW7dOp/Ph5WXISpkpebf/OY3TqcTKzVDzPDWgdh1dnZ6vd7du3ejekFUMjMz3333XZ7n7XY77SwgYejAIHZFRUXXrl3DyssQA6/Xu27dut/97nc2m412FpAqdGAQI4fD0dvbu2PHDqxqdvQAABcNSURBVFQviAHHccXFxT09PZjUDDFDAYMYkbWdMHoeYkbePB9//DHtICBVOIUIsRgZGWFZ1mAwfPvtt7SzgIRt3rzZ4/EIgrB8+XLaWUB60IFBLM6ePTs6OlpbW0s7CEjbkSNHRkdHsVIzxAYdGERtbGyMZVmdTnf37l1M/4KFIJOax8fHvV4vVmqGaKEDg6hdunTJ7/eXlpaiesECkZWah4eHv/zyS9pZQHrQgUHUNm3axPO81+vNysqinQUkz+/3syybk5OD66kQLXRgEJ2Ojg6Hw1FSUoLqBXGRlZVVUlLS39+PlZohWujAIDomk6m3t9fpdOr1etpZQCZ4ns/Pz9++ffuVK1doZwEpQQcGUXC5XD09PW+99RaqF8SRXq9/8803u7u73W437SwgJShgEIWTJ0+GQqEjR47QDgJyc+TIkVAodPLkSdpBQEpwChEi9fDhQ71en5eXd/PmTdpZQIY2b948ODjo9XpXrlxJOwtIAzowiNRnn30mimJVVRXtICBPVVVVoih+9tlntIOAZKADg4gEg0G9Xh8KhXie1+l0tOOADImiyHGcWq3meV6r1dKOAxKADgwi0tLS4vP5qqqqUL0gQXQ6XVVVlc/na21tpZ0FpAEdGMwvFApt2LDh+++/x8rLkFBkpeY1a9bcvn1bpVLRjgOpDh0YzK+zs9PlcpnNZlQvSKjMzMw9e/a4XC6s1AyRQAGD+dXV1anVaqvVSjsIyJ/ValWr1XV1dbSDgASggME8HA6H3W43mUxYeRmSQK/Xm0ymzs5OrNQM80IBg3nU19czDIP2C5LGYrEwDHP69GnaQSDVYRAHzMXn8xkMBkxehiQjk5p5nl+1ahXtLJC60IHBXBobG0VRJN+IAZLGYrGIotjY2Eg7CKQ0dGAwq2fPnpGVlzGxFJKMTJwXRVEQhCVLltCOAykKHRjM6sKFC4FAoLS0FNULkkyr1ZaWlgYCgba2NtpZIHWhA4OZkcnLQ0NDXq93xYoVtOOA4gwPD3Mcx7LsrVu3MKkZZoQODGbW0dHhcrn27duH6gVUrFixYu/evXfu3MFKzTAbFDCYWX19PSYvA12HDh1Sq9VkIgfAdChgMIPBwcGuri6j0YjJy0ARx3FFRUV2u/3u3bu0s0AqQgGDGZw4cQIrL0MqICs1nzhxgnYQSEUYxAFTkZWX8/Pzv/nmG9pZAJitW7c6nU5Maobp0IHBVE1NTaIoVldX0w4CwDA/rdR87tw52kEg5aADg18YHR3lOE6r1Xo8Ho1GQzsOADM+Pm4wGILBoNfrTUtLox0HUgg6MPiFlpYWv99fXl6O6gUpQqPRlJWV+f1+rNQMU6ADg59NTEzk5+f7fD5BELB2JaSOJ0+eZGdnr1692uFwqNX42g3/hLcC/Mxms/E8j5WXIdVkZGSYzWaPx9PR0UE7C6QQdGDws6KiomvXrg0MDGD6F6Qar9e7bt263/3udzabjXYWSBXowOCfvvvuu97eXqy8DKmJ4ziTydTT09Pf3087C6QKFDD4p7q6OgYrL0MKI+vSkTcqAINTiEA8evSI4zi9Xt/X14c7f0NqCoVCGzdu5Hne6/WuXLmSdhygDx0YMAzDNDQ0iKL4l7/8BdULUpZKpaqtrRVF8ezZs7SzQEpABwbM8+fPs7Ozf/3rX9+9exfTvyCVkUnNZKVmTGoGdGDAXLx4MRAIHDhwANULUpxGozlw4MDIyMjFixdpZwH60IEBs3HjRkEQBEFYtmwZ7SwA83j8+DHLsizL9vX10c4ClKEDU7rOzk6n0/n++++jeoEkLFu2bN++fU6n0263084ClKGAKV1dXZ1ara6pqaEdBCBSNTU1arUa4+kBBUzRPB5PV1dXUVERy7K0swBEiuO43//+93a7ned52lmAJhQwRaurq5uYmCDzQwEkxGKxTExMoAlTOAziUC4yeTkvL+/mzZu0swBE7fXXX3e73ZjUrGTowJSLrLyMe0eBRFksFqzUrHDowBRqbGyM4ziNRoOVl0GiyKTm8fFxr9e7ePFi2nGAAnRgCnX+/Pnh4WGsvAzSRVZqHh4e/uKLL2hnATrQgSlRKBTasGHDDz/8gJWXQdLISs1r1669desWbuOpQOjAlKi3t9flcu3evRvVCyQtIyNj9+7dd+7cuXr1Ku0sQAEKmBKdOnVKpVJh9DzIgMViUalUp06doh0EKEABUxyHw9He3l5cXJyTk0M7C8BC5ebmFhcXX7lyxeFw0M4CyYYCpjj19fXMT4vbAsgAuRHa6dOnaQeBZMMgDmUZHh7mOC47O/v27du46A3yEAqF1q9ff+/ePa/Xm5WVRTsOJA86MGVpbGwcGxuzWq2oXiAbKpXKarWOjo42NjbSzgJJhQ5MQUZHR1mW1Wg0PM9rtVracQDiRhRFg8Hw4sULr9eLlZqVAx2YgrS0tIyMjJSXl6N6gczodLqysjK/39/a2ko7CyQPOjClmJiYKCgouH//viAImZmZtOMAxFkgEGBZdvXq1Q6HQ63GV3NFwMusFJ2dnR6Px2w2o3qBLGVmZu7Zs8fj8WClZuVAAVMKsvIy7j0PMma1WrFSs6KggCmCw+Ho6uoqLi7mOI52FoBE0ev1JpPJbrc7nU7aWSAZUMAUgczxRPsFskfe5JjUrBAYxCF/Dx8+1Ov1BQUF169fp50FIOH+9V//1eVy8Ty/atUq2lkgsdCByR9Zebm6upp2EIBksFqtWKlZIdCByZwoinq9XqVSYfIyKASZ1BwKhXie1+l0tONAAqEDk7m///3vDx8+rK6uRvUChdDpdFVVVQ8fPvz8889pZ4HEQgcmZ+HJy0NDQ1i7EpSDrNSMSc2yh5dWzrq6ujwez7vvvovqBYqSkZHxxz/+0ePxdHd3084CCYQCJmdYeRkUCys1KwEKmGx99913dru9uLg4Ly+PdhaAZHvllVdMJlNnZ2d/fz/tLJAoKGCyRW6og/YLFIu8+XFnKRnDIA55un//vsFgePXVV2/evEk7CwA1mzdvdrvdPM+/9NJLtLNA/KEDk6empqbx8XG0X6BwFoslGAxipWa5QgcmQ8+ePWNZNi0tzePxaDQa2nEAqAkGg3q9XhRFQRCWLFlCOw7EGTowGfr8888DgUBZWRmqFyicVqstLy8PBAItLS20s0D8oQOTm1AotH79+nv37nm93qysLNpxACjz+/0sy2ZnZ9++fVulUtGOA/GEDkxu2tvb3W73vn37UL0AGIbJysrat2+fy+Wy2Wy0s0CcoYDJTX19vVqtxvANgDCyUnN9fT3tIBBnKGCy4na7e3p6jEYjVl4GCOM4rqioqLu7e3BwkHYWiCcUMFmpr68PhUKHDh2iHQQgtRw6dCgUCqEJkxkM4pAPrLwMMIetW7c6nU6v17ty5UraWSA+0IHJx7lz57DyMsBsqqursVKzzKADkwkyYROr0ALMRhRFjuPUajVWJ5cNdGAy0dbW5vP5ysrKUL0AZqTT6crKynw+38WLF2lngfhAByYHoVDotddeu3fvHlZeBphDIBBgWXbt2rW3bt3CpGYZQAcmB11dXXfu3CkpKUH1AphDZmam2Wy+c+cOVmqWBxQwOairq8PkZYBIkEnNWCRMHlDAJM/pdHZ0dBQXF2PyMsC8OI4zmUwdHR1ut5t2FlgoFDDJO3PmDMMwVquVdhAAabBaraFQCE2YDGAQh7SNjIxwHMeybF9fH+0sANIQCoU2bdokCIIgCMuWLaMdB2KHDkzampqanj9/jvYLIHIqlcpisTx//rypqYl2FlgQdGASNjo6ynGcVqvFyssAURkfHzcYDMFg0Ov1pqWl0Y4DMUIHJmEtLS1+v//gwYOoXgBR0Wg0Bw4c8Pv9ra2ttLNA7NCBSRWZvCwIgtfrXbFiBe04ABLz6NEjjuP0en1fXx8mNUsUOjCpIpOXd+/ejeoFEIOVK1fu3r17YGAAk5qlCx2YVJlMpt7eXqfTqdfraWcBkCSPx1NQULB9+/YrV67QzgKxQAcmSQ6Ho7u722g0onoBxMxgMBQVFXV1dTkcDtpZIBYoYJJEFpbF6HmABSIH0enTp2kHgVjgFKL0kJWX8/Lybt68STsLgORt3rx5cHCQ5/lVq1bRzgLRQQcmPU1NTaIoov0CiAur1YqVmiUKHZjEYPIyQHxhUrN0oQOTmNbWVr/fX15ejuoFEBcajaasrMzv958/f552FogOOjApmZiYKCgoePDggSAIWLsSIF6ePHnCsuzLL798+/ZtTGqWEHRgUmK32z0ez969e1G9AOIoIyOjpKTE5XLZ7XbaWSAKKGBScubMGbVaXVlZSTsIgNxUVlaqVCqyuh5IBQqYZDgcDpvNtmPHDoPBQDsLgNzk5OTs2LGjvb0dk5olBAVMMsgCshaLhXYQAHkiBxe5SwBIAgZxSMPw8DDHcRzH4c7ZAAmCFR4kBx2YNDQ2No6NjVksFlQvgAQhKzWPjY01NjbSzgIRQQcmAc+ePWNZNi0tDZOXARIqGAzq9XpRFAVBWLJkCe04MA90YBLQ2toaCATKyspQvQASSqvVlpeXBwIBTGqWBHRgqW5iYiI/P//BgwdDQ0OY/gWQaIFAgGXZ1atXOxwOtRpf8VMaXp5UZ7PZeJ4vKSlB9QJIgszMTLPZ7PF4Ojo6aGeBeaADS3VGo/Hq1asDAwMcx9HOAqAIXq933bp1hYWF7e3ttLPAXNCBpbT+/v6enp4dO3agegEkDcdxxcXF3d3dmNSc4lDAUhpWXgagghx0mNSc4nAKMXX5fD6DwVBQUHD9+nXaWQAU54033nA4HFipOZWhA0tdZOXlmpoa2kEAlKi6uloUxaamJtpBYFbowFKUKIocx6nVap7ntVot7TgAikMmNU9MTHi9Xp1ORzsOzAAdWIr64osvHj16VFlZieoFQIVWq62oqHj06NGFCxdoZ4GZoQNLRaFQaP369f/7v/87NDS0dOlS2nEAFOr//u//srOzsVJzylpEOwD87MWLF/v378/JycnJyXG73RUVFaheABQtXbrUbDZ//PHHX3755eDgoMfj+fTTTxctwsdmqkAHlkLu3btHFqtUq9UTExNffvnlH/7wB9qhABTt8uXL77zzDjkkGYbheX7NmjW0Q8E/4RpYClm1ahU5TUEOlXfeeWf//v20QwEo15/+9Kd33nmH+emQVKlUK1eupB0KfoYClkIWL148+ZxhKBS6fv36jz/+SDESgGK9ePFiyhTMjIyMxYsX08oD06GApZbJUyb/7d/+7R//+MevfvUrinkAFGvRokX/+Mc/3nrrrfBPMKM51aCApRZSrn71q1999NFHX3/9Nc5XAFC0atWqr7/++r/+67/CBybtRPALv/rP//xP2hngZxqN5vvvv79w4cKuXbswbBeAOpVKtXXr1u3bt/f19VVXV2/YsIF2IvgZRiECAIAk4RQiAABIEgoYAABIEgoYAABIEgoYAABIklIKWH19vTYCSV6Atb6+vrKyMpmPOK8UjAQwN0EQyLoNYZWVlYIgxLxDHAVSoZQCloJsNlttbS3tFL+QgpEA5rZt27a8vLzm5ubJP2xubs7Ly4utCOEokBClFLCamprgL5GfT/khlj8GkJBt27bduHGDYZjjx4+73W5yFLvd7uPHjzMM09zcvG3bNtoZIYGUUsAAQGYqKytv3LixZcsW8tWTZVnyc5Zla2pq3G73li1bbty4gZOBMoYCNgObzUauh5F/aLXabdu2hU+pTznhPvl/TdnJtm3bJm82+QLbtm3bdu7cyTBMc3MzOWXPMAzZrc1mEwQh/Lvh3wpfxtu2bZvNZpvycPOmCu98crDJlwpmjASQCENDQ06ncyF7EASBnDa8du3ajBuwLPvpp58yDNPc3Bx+k5OjYMqlbnJkkTf8HEfB5ANnxmOQmXYYTr8Ul9BjXIFQwGbldrvJW5kg3+8EQZhywv3GjRt5eXlT3m2VlZU7d+4kJzfCm9XW1kZSFXiez8vLC/9ubW0tuaQcPi9/48aNnTt3Tn7vRpiKYZiPPvpocjByqWDeSADxtW/fvk2bNu3fv//Bgwex7eHKlSsMw5BThbNhWba0tDS88ULU19dPPnDIMTilENpstimHITm+ph+GCT3GFQUFbFbNzc3k7EQwGAx/yyMf96WlpeHLZuQQmlzqbDYbeZ9dvnw5vNnly5eZSV8Gr127Rn5CdtXQ0BD+9dra2vD+yc5ra2ubm5vJ3siJEeaXh24kqYgbN26ENyMBSOC5IwHE19OnT0Oh0Oeff56Xl3fs2LHR0dFo93Dp0iWGYfR6/dybkUOAbByJGY8CQRBIaTl+/PjkY2fyWA9BEMhjlZaWhq/GkfI5pRQxCT7GFQVrY8+FnIIII1+4SktLJ3+4k3Ef5DsU+Td5cx8/ftxoNIY3MxqN5Iw8z/Phk/Uz2rJlS3j/xcXF4SOH7I1l2b/+9a87d+68c+dOVKmm79xoNJaWljY3N/M8PzlqbP7whz94PJ4F7gQU4v79++Qfo6OjH3744blz5+7evavT6SLfA2lf5i1gZIPJ50JiQBq40tLS8KEUPnZsNhs5dsLbTD4Myb+bm5uPHz8++ecJPcYVBQVsVlu2bJlSacj3uOlDbMlb8NKlS+Rt1NDQEH6fCYLA8zzP85cuXYrwKCIrwBLhAMXFxbNtH2Eq4tVXX528Dfla53a7Iwk2t/T09IyMjIXvB5TA5/NN+UlsC7fO/V0wXsghNqXXmXyYz7YN+Ulzc3O4FBEJPcYVBQUsarNdNJpcnwRB2L9//wK/90UlklRzbLZwra2tCdozyM/69etdLhfDML/+9a//4z/+48iRI2lpaTHsRxCEJNSwSLq92bYhTVVcPgoiPMYVBdfAZjWlWWEifqOQy63hjbds2UImqZDz2vOa96xIbKkAUkd6erpKpTKbzS6X64MPPoihepGjief5uTcjG0R46M0tkkoZYTXFMR4v6MCi5na7536b7t+/n5l2wjrR5k0FkDr++7//+9mzZ/n5+THv4Z133iFXlOe+fEsuSE8+ZRezSLq9hHaEOManQwcWhQi/9JGvS9NPWCfoa1SEqQBSR3Z29kKqF/PTFaO57/kUnis2x+WlSERyiM22DRniu8AWEMf4bFDAokC+x3300UdTfk7mO89905rwdI24vwsXkgpAosJzvGabz0uuQzMMU1paGm5cZhy1NO8ge3KIhaedEJOnP8+2Tfgn069HRAXH+GxQwKJAhvrcuHFj8jFjs9nI0KO//vWv5CfT53CQWZAz7nPK8KTEpYrcwiMBJEFDQwOZmpKXl1dfXz/5Xjn19fXkOvTkAevMTxefyPB3smX4bopTTD4KSAPX3NwcnrkcvuFv+LgObxO++wa5cQZpARd4d+C4H+OygQIWHfJ9ihwz5IYu4dmL4XPx5P1EbkVD1NbWkqEczKRvf+EZKgv/DhVJqkjEMRJAEly7do30YbW1teE3f15eHikYpaWlU240RaZjMgyzc+dOsuWNGzemtE3TjwKWZcPTjWc7vliWDd+sgCQJ3zjj8uXLC792Fa9jXGZQwKJjNBrdbjc5ZogtW7Zcvnx58rc8o9F4+fLl8FlvssG1a9fIgRH+Zhc+KpKTKhJxjASQHA0NDVPe/MxPt8OY8f0frnkMw2zZsmX6JMgZj4KamprpB/WU/U8/DEmMuBSYeB3jMqMKLywCAAAgIejAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAklDAAABAkv4fmhFdR5p/D98AAAAASUVORK5CYII=)

---

# Random experiments and causality

- The whole idea of running an experiment is to *add another source of variation in the Treatment that has no back doors*

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAIAAADOgk3lAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dXUwU6Z7H8WrAVlobx4PM+Da+gDOCLxwURPodMxfjzsm6Xpy4yeZcnXizk5jZjDnJGi/MXJjZG3ezMfEke8hJzs1md9ZkJya70YsTpF8BFRQBFWVWWFlGPfgCCoINvRfPmZ6ebmiqu6v66ar+fi4m2FNU/bu6qF/X8zz1lGVubk4BAMBoSmQXAABANggwAIAhEWAAAEMiwAAAhkSAAQAMiQADABgSAQYAMCQCDABgSAQYAMCQCDAAgCERYAAAQyLAAACGRIABAAyJAAMAGBIBBgAwJAIMAGBIBBgAwJAIMACAIRFgAABDIsAAAIZEgAEADIkAAwAYEgEGADAkAgwAYEgEGADAkAgwAIAhEWAAAEMiwIBiVFlZabVajx8/LrsQVY4fP65Hte3t7dblnDt3LsetnDt3zmq11tfXa1KzXKkfxPHjxyW+NQIMKDrt7e1TU1OKovzxj3+UXQsM7Ny5c99++63EAsokbhuAFF988YWiKLW1tffu3Wtvbz98+LDsipbxzTff6Ldyu90+MTGh3/rNRNcPIgtcgQFF5969e3a7/a//+q+VH8IMMCICDCguolPnk08+OXPmjKIo9+7dS12mvr5edP+cO3dO9JaJXpz29vaMlhHa29vFwosuI7qIKisrk35LdLeI/pWkrhf1m45vt7Kysr29PZe+NPUbFe8ofRda+n2SUf1JqxJLZlf5sqtKLKCysvKrr75SFOXevXtihWo+So3NASgmtbW1iqJcvXo1/vPZs2cXXcZut6c5Y6hZZm5u7uzZs4ueeRI3uui5SKxZLHbs2DFFUY4dO5bRplMXEL8YX8/Vq1fFetTvt2U3KhZL3WhtbW1G+0RN/fG3kH5VKitXs6rEDyJpheINpq426aPUFldgQBFpb28X7Yei30u0Iv7jP/7jogtPTU3FT7vxc27SJUX6Zdrb28WX9NraWhGZ8fPpV199Ff92L15JXbOiKOIyMYvyjh8/LtYgzptXr1612+2LXm5mJP1Gz507JzYRP1mLjsbENajcJyrr//TTT5WfpqPIGLEJ9ZVntCphYmJCrEf8Sl9fn5LtR5k1AgwoIr/97W8VRfnkk0/EP8U5ZWpqatH2K7vdLs5KYklxbrp9+7b6ZcTmamtr+/r64kNF+vr6xGLx7jeRo//+7/8eX604A6ZezajftBhgefbsWfEeDx8+fOnSpUXXMzU1tdQY+tTWsPQbFW/h2LFj8ZN1X19f0pWKyn2ipn7RlCdWFX/xm2++EcGT1NKYvvKMVpVGdh9l1ggwoIiI02LiWDJxZhFn1SSbN29O/KdYMukiIP0yYnPipJZIvDI2Nib+mdobJ86Aqb+octPiPgG73Z74rf/w4cO5n0bTv1/xw9/+7d8mLhP/uiCo2Scq6xer+ud//uekVYkCku6RUPNJqVxVGtl9lFkjwIBiIU6LSSdBcWZZ9G4eNaf79MuItiOn05n0evzKL2k98aYncQZM3+iUZtPhcFhJOWUv9Stp+sBSh9er2SdJtyX8/Oc/T/ynmn2SUf2ffvpp0oWjaAxM3L0qK1e5qvSy+CizRoABxUJcZokxY3HxHo7cp5xYipr7zBKbnnRtdNLPos2wS9Hk3ruMciVvq8rnR0mAAcUi/aQJif0W2lJzZhff0EUDmlaNTvEmyvzIKJPU7BOV9cdHgqRSX4+Gq9Ljo1wKAQYUhfh34dRzkxg/LWbl0HajYvyCaBBLLSZ1HLYYTiLOfbk0OqU20Am5j0JUI2k3pg57UZbbJyrrX2pVWdBwVYqmH2V6BBhQFNJ8Fz58+LA4fy06lCMXYvxC6rWdeCWpjyc+M0hqR12mxDuamppKahfVO8AWHRGTNAJCzT5RWb9YVepdEEvdUJyGhqtSNP0o0yPAgKKQvi9dnL80n9tXjGG7d+9efN4HMdeDKCZpzFviALbcz3riHcXvrGpvb8/0LJwF8Y6+/fbbxElDkrqXVO4TNfWL0aRTU1OJU2acO3dO9Gt++eWX6ivPZVWpTZ3afpRpEGCA+cXv8llqAXFWTf3Kn6PDhw+Le13v3bsnRrh9+umn8Vt9UzuN4o2KSSPRs/DNN9+ItcW3u9TVQJr7wDK9+Dh8+LC4cerbb78Vv37v3r2kjarcJyrrF6uampqKDyCM3yWdacNdFqsSTZ1iBybtKA0/yjQIMMD8lrr3KC7eiqj5UI4zZ85cvXo18cwrZqBY9JwoLjviE4XkaGJiIr5du92+1FRJ2vrmm2/Onj0bP32fPXs2dber3Cdq6k9dld1uP3v2bOL9yCplsap4YKfS9qNciiWLkSoAoIfjx49/++23x44d0+mxHaKlLj69heEYqH69P0qBKzAAhUJcKebe6BR/1HLS66KlbqkxfoXD6PUr2n2U6RFgAArCuXPnRDdP7o1O8TUkPsJD/Kx3o5YmjF6/hh9lejyRGYBk8TF4ymLT8WXn7NmzX331lZh2JPH1pab0LTQGrV+PjzINrsAASBYfOLDo0MTsnDlzJnEwhfLDQAn163/y5Mns7Gzq6//7v/+rSYXp5V6/FHp8lGkwiAMAFvE3f/M3//Vf//Xo0aN169bFX/ziiy9++9vffvfdd1u2bJFYGwSuwAAgWSwW6+jo2LZtW2J6KYrS0NCgKEpHR4ekuvATBBgAJLt///6zZ8+8Xm/S6+IVv98voygkI8AAIFkoFFIUxeVyJb1eU1OzYcMG8X8hHQEGAMkCgYCiKB6PJ/V/ud3uoaGh77//Pu9FIRkBBgDJOjo6duzYsehIDdGKGAwG814UkhFgAPATjx49GhsbW/TyS/nhsowAKwQEGAD8hHiu41IzNtXV1a1bt45usEJAgAHAT6TpAFMUpaSkxOl03rlz5+XLl/mtC8kIMAD4iWvXrm3atOmjjz5aagGv17uwsEAronQEGAD86P/+7/+Gh4eXuvwS6AYrEAQYAPxIdICl3gGWqKGhwW63i5ZGSESAAcCPxDRRPp8vzTJlZWUOh6O3t3dqaipfdWERBBgA/CgYDL7//vvxWdWX4na7o9FoZ2dnfqrCoggwAPizFy9e3L17t7m52WKxpF9StDGK9kbIQoABwJ+Fw+GFhQW3273skgcPHly1ahXdYHIRYADwZ2Ka+fQdYMKqVasOHTrU1dU1MzOjf11YHAEGAH8WCAQqKirEQ7+W5fV6Z2dnu7u79a4KSyHAAEBRFGVycvLWrVtOp7O0tFTN8mKuKeaUkogAAwBFUZRIJBKNRtPfAZbI6XRarVYebikRAQYAivLDtVT6OTgSlZeXHzhwoLOzc25uTs+6sCQCDAAURVECgYDNZmtqalL/Kx6PZ3p6uqenR7+qkAYBBgDK27dvb9682dTUZLVa1f+WaG9kML0sBBgAKN3d3W/fvlUzgD6Ry+UqLS1lVl9ZCDAA+PPU8uo7wIS1a9fu3bs3FArNz8/rUxfSIcAAQAmHwytWrDh48GCmv+h0OicnJwcGBvSoCukRYACK3bt370KhUENDw+rVqzP9XdHqyKSIUhBgAIrdzZs337x54/V6s/hdr9drsVjEQ1iQZwQYgGInhhGqmcM31fr163ft2hUIBGKxmNZ1YRkEGIBiFwqFSktLswswRVFcLtfTp08fPnyobVVYFgEGoKjNz8+HQqG9e/euXbs2uzWItkdaEfOPAANQ1Pr7+1+9epXpAPpEIsC4Gyz/CDAARU0ET9bth4qibN68efv27VyB5R8BBqCoiTl8xbNRsub1esfGxkZGRjQqCqoQYACKVywW6+joqK2t3bBhQy7rERdwPFolzwgwAMVraGjo2bNnuXSACWINdIPlGQEGoHiJyFH/EMul1NTUbNiwgacz5xkBBqB4iVuYc78CUxTF7XY/ePDgyZMnua8KKhFgAIpXR0dHdXX1hx9+mPuqvF5vLBbj2WD5RIABKFL/8z//MzY2psnll/LDOA66wfKJAANQpHKZAjHV7t2733vvPaalzycCDECRyu4hlkspKSlxu919fX0TExOarBDLIsAAFKlAILBly5bq6mqtVuj1ehcWFiKRiFYrRHoEGIBiND4+Pjw87HA4NFynGI5PN1jeEGAAipG4Z0ur9kNh//79drudgYh5Q4ABKEZi2qfsnsK8lLKyMqfT2dvbOzU1peFqsRQCDEAxCgaDVVVVdXV12q7W5XJFo1G6wfKDAANQdF6+fDk4ONjc3GyxWLRds+gGI8DygwADUHRCodDCwoK2HWDCwYMHV61aRTdYfhBgAIqO6ADz+Xyar3nVqlWHDh3q6uqamZnRfOVIQoABKDqBQKCioqKhoUGPlXu93tnZ2e7ubj1WjkQEGIDiMjU1devWLYfDUVpaqsf6xb1lPFolDwgwAMUlEolEo1GtpkBM5XQ6rVYr3WB5QIABKC7aToGYymazNTY2RiKRubk5nTYBgQADUFxCodCqVasaGxv124TH45menu7p6dFvE1AIMABF5e3bt9evX29ubl65cqV+WxF3g9GKqDcCDEAR6erqevv2rR4D6BO5XK7S0lLGceiNAANQRMQdYPqN4BAqKirq6+tDodD8/LyuGypyBBiAIhIOh1esWNHc3Kz3hlwu16tXrwYGBvTeUDEjwAAUi7m5uXA43NTUtHr1ar23JVopOzo69N5QMSPAABSLmzdvzszM6DeAPpHL5bJYLDzcUlcEGIBiIYYFiiGCelu/fv2uXbvC4XAsFsvD5ooTAQagWIRCodLS0vwEmKIoTqfzyZMnw8PD+dlcESLAABSF+fn5UCi0b9++ioqK/GyRbjC9EWAAisLt27cnJyfz0wEmeL1ehduZ9USAASgK4XBYURSn05m3LW7evHn79u1cgemHAANQFCKRiMViyecVmKIoHo9nbGxsZGQknxstHgQYAPOLxWIdHR11dXXvv/9+Prcr8pJWRJ0QYADMb2ho6OnTp3rPIJWKANMVAQbA/MS8uvnsABNqamo2bNjArL46IcAAmJ8YSdHa2pr/TXs8nqGhofHx8fxv2vQIMADm5/f7d+zYsWnTpvxvWrRbijGQ0BYBBsDkHj16NDY2lufxh3Fiu0yKqAcCDIDJiTEUsgJs9+7d7733Ht1geiDAAJic3AArKSlxu919fX3Pnz+XUoCJEWAATC4QCGzZsqW6ulpWAW63e2FhIRKJyCrArAgwAGb2/fffDw8POxwOiTXQDaYTAgyAmYnOp/zfwpxo//79drud25k1R4ABMDO/36/88GQTWcrKylpaWnp6eqampiSWYT4EGAAzCwQClZWVdXV1cstwu93RaLSzs1NuGSZDgAEwrZcvXw4ODra0tFgsFrmViMdAM45DWwQYANMKBoMLCwty2w+FQ4cOlZeXX7t2TXYhpkKAATAt0QEmnows18qVK5ubm69fvz4zMyO7FvMgwACYVigUstvt9fX1sgtRFEVxuVyzs7M3b96UXYh5EGAAzOn169e9vb0Oh6OsrEx2LYryw1B+BtNriAADYE6hUCgajRZC+6HgdDqtVisBpiECDIA5FcItzIlsNtv+/fsjkci7d+9k12ISBBgAcwqFQqtWrWpsbJRdyI+cTuebN2/6+vpkF2ISBBgAE3r79u3169cPHjy4cuVK2bX8SEyKSCuiVggwACbU1dX19u3b1tZW2YX8hMfjKS0t7ejokF2ISRBgAExIXOUUTgeYsHbt2n379gWDwfn5edm1mAEBBsCEwuHwihUrmpubZReSzO12v3r1amBgQHYhZkCAATCbubm5cDjc1NS0evVq2bUkE8P6xRQhyBEBBsBsenp6pqenC+cOsEQul8tisfBwS00QYADMRlzfiAngC01VVdXOnTuZll4TBBgAswmFQqWlpU6nU3Yhi/N4POPj4w8fPpRdiOERYABMZX5+PhQK/fznP6+oqJBdy+JE2yaD6XNHgAEwlb6+vsnJSXHLcGESzyejGyx3BBgAUxFTIBZmB5iwefPm7du3cwWWOwIMgKn4/X6LxVJotzAn8Xg8jx8/HhkZkV2IsRFgAMwjFosFg8Ha2tr169fLriUdng2mCQIMgHkMDQ396U9/KuQOMEFUSDdYjggwAOYhOsAKdgB9XE1NzYYNG0S1yBoBBsA8xMiIQpuEPpXFYvF6vffv3x8fH5ddi4ERYADMw+/379ixY9OmTbILWZ5oRWRKjlwQYABM4tGjR2NjY4XfASaIgf60IuaCAANgEoX5DLCl7N69+7333rt27ZrsQgyMAANgEmJQX2FOQp+qpKTE5XL19/dPTEzIrsWoCDAAJuH3+7ds2VJdXS27ELU8Hk8sFguHw7ILMSoCDIAZfP/998PDwy0tLbILyQB3g+WIAANgBqIDTMyTaxT79++32+0EWNYIMABmIGLAKCM4hLKyspaWlt7e3qmpKdm1GBIBBsAMgsFgZWXl7t27ZReSGZfLFY1Gu7q6ZBdiSAQYAMN7+fLl4OBgS0uLxWKRXUtmxCUj4ziyQ4ABMLxgMDg/P2+UAfSJDh06tGrVKp4Nlh0CDIDhiQ4wo8zBkWjlypXNzc3d3d0zMzOyazEeAgyA4QUCAbvd3tDQILuQbLjd7tnZ2Zs3b8ouxHgIMADGNjU11dvb63A4ysrKZNeSDR5umTUCDICxdXZ2RqNRYw2gT+R0Oq1Wq9/vl12I8RBgAIzNuB1ggs1mO3DgQCQSmZubk12LwRBgAIwtFAqtWrWqsbFRdiHZ83g809PTvb29sgsxGAIMgIHNzMx0dXWJweiya8meeDYY3WCZIsAAGFh3d/fs7KwR7wBL5HK5SktLebhlpggwAAYmTvoOh0N2ITlZu3btnj17wuHwwsKC7FqMhAADYGDBYHDFihVGDzBFUTwez4sXLwYGBmQXYiQEGACjmpubC4fDjY2Nq1evll1LrsSDYJhTKiMEGACj6u3tnZ6eNu4A+kQul8tisfBssIwQYACMSkzi7nQ6ZReigaqqqp07d0YiEdmFGAkBBsCoAoFASUmJOQJMURSPxzM+Pv7gwQPZhRgGAQbAkObn5wOBQH19/bp162TXoo3W1laFbrBMEGAADOnOnTuvXr0y+h1gicR74XZm9QgwAIYk7gATc1iYw6ZNm7Zt28btzOoRYAAMqaOjw2KxmGMIYpzP5xsdHX306JHsQoyBAANgPLFYLBQK1dbWrl+/XnYtWhJ5TCuiSgQYAON5+PDhs2fPTDP+MI4AywgBBsB4xCneuA+xXEp1dfWWLVsIMJUIMADGI8aai3HnJuNyuYaHh8fHx2UXYgAEGADj8fv927dv37x5s+xCtEcronoEGACDefTo0djYmMnGH8aJGwMYTK8GAQbAYMzaASbs3r37/fff9/v9sgsxAAIMgMGIADPrFZjFYmlpaRkcHJyYmJBdS6EjwAAYjN/v37hx486dO2UXohe32x2LxZiZflkEGAAjefz48XfffWfW9kOBbjCVCDAARiJO6+YOsAMHDlRUVDAt/bIIMABGYu4OMKG0tNThcNy6dWtqakp2LQWNAANgJOFweN26dbt375ZdiL5cLlc0GqUbLD0CDIBhvHz5cnBw0OPxlJSY/NwlLjHpBkvP5AcBADMJBoMLCws+n092Ibo7ePCgzWZjPo70CDAAhmHuW5gTWa3W5ubm69evz8zMyK6lcBFgAAwjGAza7fZ9+/bJLiQfnE7n7OzszZs3ZRdSuAgwAMYwOTnZ09PjcrnKyspk15IPXq9X+WHefSyKAANgDJFIZH5+XtzkWwxaWlqsVivjONIgwAAYgziVi+uSYmCz2RoaGjo7O6PRqOxaChQBBsAY/H6/zWZramqSXUj+OJ3O169f9/X1yS6kQBFgAAxgenr6xo0bhw4dWrFihexa8kfcMHDt2jXZhRQoAgyAAXR3d8/NzRXDAPpETqezpKQkGAzKLqRAEWAADEB0gDmdTtmF5NW6dev27t0bCoUWFhZk11KICDAABhAKhVasWNHS0iK7kHzzeDwvXrwYGBiQXUghIsAAFLq5ublwONzY2Lh69WrZteSbmBSROaUWRYABKHS3bt2anp4utg4wwePxWCwWAmxRBBiAQhcOh5Xi6wATqqqqdu7cKfYAkhBgAApdR0dHaWmpuR9imYbP5xsfHx8aGpJdSMEhwAAUtPn5+WAwuG/fvrVr18quRQ6R3H6/X3YhBYcAA1DQBgYGXr16VZwdYIJ470yKmIoAA1DQxPiFom0/VBTlww8/3LZtG+M4UhFgAAqa3++3WCzFHGCKoni93tHR0ZGREdmFFBYCDEDhisViwWCwtrZ2/fr1smuRSbQichGWhAADULgePnz47Nmz4hxAn0g8BY3B9EkIMACFiw4w4aOPPvrggw+4AktCgAEoXB0dHYqitLa2yi5EMovF4vV679+/Pz4+LruWAkKAAShcfr9/x44dmzZtkl2IfOJR1DxaJREBBqBAjYyMjI2NFfMdYIlENxh3gyUiwAAUKNF+KJ5KjD179lRVVYl9AoEAA1CgGMGRyGKxOByOwcHBiYkJ2bUUCgIMQIEKBAJbtmzZsWOH7EIKhdPpjMVinZ2dsgspFAQYgEL0+PHj7777jg6wREyKmIQAA1CIxGmaAEt04MABu91ON1gcAQagEInx4mLsOISysjKHw9Hb2zs1NSW7loJAgAEoRMFgcP369bt27ZJdSGFxu93RaJRuMIEAA1Bwnj59Ojg46Ha7LRaL7FoKixiTye3MAgEGoOB0dnbGYjE6wFI1NTWVl5czKaJAgAEoOOIKgwBLtXLlyubm5uvXr8/MzMiuRT4CDEDBCYfDa9asqa+vl11IIXK5XLOzszdv3pRdiHwEGIDCMjk52dPT43a7y8rKZNdSiMTITL/fL7sQ+QgwAIUlEolEo1Exdy1StbS0WK1WbmdWCDAAhUacmpkCcSk2m23//v1dXV3RaFR2LZIRYAAKSyAQsNlsTU1NsgspXE6nc3Jysre3V3YhkhFgAArIzMzMjRs3mpubrVar7FoKl7g8ZU4pAgxAAenu7p6dnaX9MD2Xy1VSUkI3GAEGoICIk7LT6ZRdSEFbt27d3r17Q6HQwsKC7FpkIsAAFJBAIGC1WgmwZXk8npcvX/b398suRCYCDEChePfuXWdn5/79+8vLy2XXUujENCVFPqcUAQagUNy6devNmzd0gKkhZjou8ll9CTAAhYIOMPU++OCDmpqacDgsuxCZCDAAhaKjo6O0tJQrMJVaW1vHx8eHhoZkFyINAQagICwsLITD4b17965du1Z2LcYgkr6Yu8EIMAAFob+//8WLF1x+qSemiyzmu8EIMAAFQVxJ8Aww9bZu3bp161auwABAskAgYLFYuALLSGtr68jIyMjIiOxC5CDAAMgXi8UCgcCuXbuqqqpk12Ik4oK1aJ8NRoABkO/hw4fPnj1jAH2mxAVr0d4NRoABkE9MrN7a2iq7EIOpqanZuHFj0XaDEWAA5BOnYK/XK7sQ4/F4PA8fPhwfH5ddiAQEGAD5QqHQtm3bNm3aJLsQ4xGD6YtzSg4CDIBko6Ojo6OjjD/MTjHfzkyAAZBMdID5fD7ZhRjSnj171q9fX5wDEQkwAJKJQXTcwpwdi8XicDgGBgYmJiZk15JvBBgAycLh8IYNG2pqamQXYlROpzMWi3V1dckuJN8IMAAyPXnyZGhoSIxEQHaK9m4wAgyATH6/PxaLcQdYLg4cOGC320VXYlEhwADIJEYfcAdYLsrKyhwOR29v79TUlOxa8ooAAyBTKBSqrKysra2VXYixuVyuaDTa2dkpu5C8IsAASPPs2bOBgQG3222xWGTXYmziErbYng1GgAGQprOzMxaLMYA+d01NTeXl5cV2NxgBBkAaMQESQxBzt3LlysbGxhs3bszOzsquJX8IMADSBAKBNWvWNDQ0yC7EDLxe79u3b69fvy67kPwhwADIMTU11dPT43K5ysrKZNdiBqIbrKgG0xNgAOSIRCLRaJQOMK20tLRYrdaiup2ZAAMghzjVMgm9Vmw2W0NDQ1dXVzQalV1LnhBgAOQIBAI2m62pqUl2Iebh8/lev35948YN2YXkCQEGQIKZmZkbN240NzdbrVbZtZiHGM9ZPM8GI8AASNDd3T07O0v7obZcLldpaWnx3M5MgAGQQNwB5nQ6ZRdiKmvXrt27d28oFJqfn5ddSz4QYAAk8Pv9VquVANOcx+N59erVnTt3ZBeSDwQYgHx79+5dZ2dnY2NjeXm57FrMRtyWUCRzShFgAPLt9u3bb968YQYpPTgcDkVRIpGI7ELygQADkG9itgjaD/WwcePGXbt2iS5G0yPAAOSb3+8vKSnhCkwnPp9vfHx8aGhIdiG6I8AA5NX8/Hw4HN67d++6detk12JOohusGO4GI8AA5FV/f/+rV6/EzLPQg2ibLYa7wQgwAHklrgyYw1c/W7du3b59e3t7u+xCdEeAAcirYDBosViYg0NXHo9nbGxsZGREdiH6IsAA5E8sFguFQh9//HFVVZXsWsxMfD8wfTcYAQYgf4aHh588ecL4Q70RYACgsWvXrimK4vP5ZBdicjU1NRs3biTAAEAz4pTKEMQ88Hg8Dx8+HB8fl12IjggwAPkTCoW2bdu2efNm2YWYn2inNfeUHAQYgDwZHR0dHR1l/GF+iBsVgsGg7EJ0RIAByBMxBSIdYPmxd+/eyspKc09LT4AByBPRAcYVWH5YLBan09nf3//8+XPZteiFAAOQJ4FAYMuWLdXV1bILKRZOpzMWi3V2dsouRC8EGIB8GB8fHx4e5g6wfBKttSZuRSTAAOQD7Yf519DQUFFRQYABQE7E5OhcgeVTWVnZoUOHbt++/fr1a9m16IIAA5APwWCwsrJy9+7dsgspLm63+927d2btBiPAAOhuYmKiv7/f7XZbLBbZtRQX0Q1m1jmlCDAAuguHw7FYjGeA5V9jY2N5eTkBBgBZEhMaiScFI59WrlzZ2Nh448aN2dlZ2bVojwADoLtAILBmzZr9+/fLLjzQkkwAABK0SURBVKQYeTyet2/fXr9+XXYh2iPAAOhramqqp6fH5XKVlZXJrqUYiW4wMY+XyRBgAPQViUSi0SgD6GVpaWmxWq2mnNWXAAOgL3EHGLcwy2Kz2RoaGrq6uqLRqOxaNEaAAdBXIBCw2WwHDx6UXUjx8vl8r1+/vnnzpuxCNEaAAdDRzMzM9evXm5ubrVar7FqKl7iBwXyD6QkwADq6efPm7OwsA+jlcrlcpaWl5usGI8AA6Eh86+cWZrkqKirq6+tDodD8/LzsWrREgAHQUUdHh9Vq5QpMOo/H8+rVqzt37sguREsEGAC9zM3NdXZ2NjY22mw22bUUOzEK1GTdYAQYAL3cunVrenqaO8AKgcPhsFgs4pYG0yDAAOjl2rVriqJ4vV7ZhUB5//33a2trA4FALBaTXYtmCDAAegmFQqWlpXSAFQi32/3s2bOhoSHZhWiGAAOgi/n5+VAoVF9fX1FRIbsWKMoPY0HNNJieAAOgi76+vsnJSQbQFw7RGWmmbjACDIAuuAOs0GzdunXbtm1+v192IZohwADoIhAIWCwWAqygeDye0dHR0dFR2YVogwADoItIJLJz586qqirZheBHJpsUkQADoL2HDx8+ffq0tbVVdiH4CXE7s2nGcRBgALQnnv8rngWMwvHRRx9t3LiRKzAAWBIjOAqW2+0eGhoaHx+XXYgGCDAA2gsGg9u2bdu0aZPsQpBMfKsIh8OyC9EAAQZAY48ePRodHaX9sDCZqRuMAAOgMdoPC9mePXsqKyvN0Q1GgAHQmPh2zxy+hclisTidzv7+/ufPn8uuJVcEGACN+f3+LVu2VFdXyy4Ei3M6nQsLC52dnbILyRUBBkBL4+Pjw8PDPAOskInuSRPMKUWAAdASHWCFr6GhoaKiggADgJ8Q47O5AitkZWVlhw4dun379uvXr2XXkhMCDICWAoHAz372s927d8suBOm43e53794ZvRuMAAOgmYmJif7+frfbXVLCuaWgmaMbjIMMgGYikUgsFqMDrPA1NTXZbDaj3w1GgAHQjOgAczqdsgvBMqxWa3Nz840bN2ZmZmTXkj0CDIBmrl27VlFRceDAAdmFYHkej2d2dra7u1t2IdkjwABoY3Jy8tatWw6Ho6ysTHYtWJ4JHm5JgAHQRldXVzQaZQC9UTQ3N69YscLQ09ITYAC0IaZAFJOdo/CtXr16//79nZ2d0WhUdi1ZIsAAaMPv99tstubmZtmFQC2fz/f69euenh7ZhWSJAAOggZmZmRs3bohWKdm1QC3R3mvcbjACDIAGbt68OTs7ywB6Y2lpaSkpKTFuNxgBBkADzOFrRGLSr3A4HIvFZNeSDQIMgAY6OjqsVitXYIbT2to6MTFx584d2YVkgwADkKu5ubnOzs7GxkabzSa7FmTG0HeDEWAActXb2zs9PU37oRF5PB6LxSJugTAcAgxArsSk5twBZkRVVVUff/xxMBg0YjcYAQYgV6FQqLS0lA4wg/J6vU+ePHnw4IHsQjJGgAHIycLCQmdn5549eyoqKmTXgmyIbx6hUEh2IRkjwADkZGBg4Pnz57QfGpf47Iw4joMAA5CTjo4ORVG8Xq/sQpClrVu3bt++vb29XXYhGSPAAOQkGAxaLBaGIBqa1+sdGxt79OiR7EIyQ4AByF4sFgsEArW1tVVVVbJrQfbEpIiG6wYjwABk7/79+8+ePaMDzOh8Pp/yQ2uwgRBgALLHFIjmUF1dvXHjRsON4yDAAGRPzODACA4T8Hg8w8PD4+PjsgvJAAEGIHuhUGjbtm2bNm2SXQhyZcRJEQkwAFkaGRkZHR1tbW2VXQg0QIABKCKi/ZARHOawZ8+eyspKAgxAURCjrsUIbBidxWJpaWm5d+/e8+fPZdeiFgEGIEvXrl374IMPampqZBcCbfh8PjGzpexC1CLAAGRjbGzs4cOHtB+aibgbTDwcxxAIMADZEO2HBJiZ1NfXr127lgADYHKM4DCf0tLS5ubm27dvv3nzRnYtqhBgALIRDAYrKyv37NkjuxBoyePxvHv3ziiTIhJgADI2MTExMDDgdDotFovsWqAlcUlNgAEwrUgkEovFmALRfA4ePGiz2YxyNxgBBiBj4hs6AWY+Vqu1ubn5+vXrMzMzsmtZHgEGIGMdHR0VFRUHDhyQXQi05/F4Zmdnu7u7ZReyPAIMQGYmJydv3brlcDhKS0tl1wLtialVDNGKSIAByExXV1c0GnU6nbILgS6am5tXrFgRiURkF7I8AgxAZngGmLmtWbNm//79kUgkGo3KrmUZBBiAzHR0dNhstubmZtmFQC9er/f169c9PT2yC1kGAQYgA9PT0zdu3Dh06NCKFStk1wK9iG4wcaldyAgwABno7u6em5tjAL25ORyOkpKScDgsu5BlEGAAMiAGpxFg5vazn/1s7969gUBgYWFBdi3pEGAAMhAKhVauXNnS0iK7EOjL6/W+ePGiv79fdiHpEGAA1Hr37l1XV1dDQ0N5ebnsWqAvQ3SDEWAA1Ort7X3z5g2PUCkGbrfbYrEU+O3MBBgAtcSjDgmwYvDBBx/s2rXL7/fHYjHZtSypTHYBAArd06dP/+Vf/mXnzp1//OMfS0tLReMSTM/j8fzud7/7t3/7t5GRkfr6+s8++0x2Rcksc3NzsmsAUNC+/fbb48ePi5+tVuuvfvWrv/iLv/irv/oruVVBP6Ojo3/4wx8uXbp09+5d8YrD4ejo6JBbVSoCDMAyxsfHt2/fntiUVFZW9qc//clms0msCvr5u7/7u4sXLya+8uWXX/7DP/yDrHqWQh8YgGVs3LixoaFBPHxZ/Pfrr78mvUzs1KlTW7duTXzlyJEjsopJgwADsLzPPvtMXIHFYrGzZ89+8cUXsiuCjj788MOrV69u2LBB/HPNmjWF2fFJgAFYXvwL+G9+85szZ87ILQZ5UFNTc/Xq1TVr1iiKsm/fvsKc+pI+MADLW1hYOHbsWE1NzT/90z/JrgX5EwqFTp06de7cuU8++UR2LYsgwAAAhkQTIgDAkAgwAIAhEWAAAEMiwAAAhpR9gFmX5vP5Pv/8cw2rVOPKlStWqzU/271w4YLVar1w4YIea058C/ptCIVGfNbLyvPBkHRAFoICLCnPhoeHP//888Sj4vPPPx8eHs56hcbdpbpcgUUikba2NqvVqsfKTezKlSunTp2SXQXwZwV4QBZgSXnm8/nq6ura2toSX2xra6urq8suhAy9S3MNsLnFXL58Wfxfg6b6sk6ePDk3N3fy5EnTbAjSic86kXg96UUOhmLm8/kikYiiKOfPn7979644JO7evXv+/HlFUdra2nw+n+wa80qXK7AjR46IOYyTviYAALLz+eefRyIRh8MhvsfU1NSI12tqak6ePHn37l2HwxGJRMx62bAovQZxxHdukitXrvh8vsTesqQ2fdG2e+XKlcQlF23hFZ1ewpUrVxbdXFJjcep64psbHh6Oby5eUrxbwufzJW4iqWsqTY9F4ubSv3efz3f06FFFUUTrqzgKF+0DU/+mlt2H0NDQ0FD82RN6Ewf/hQsX4n8FPp8v/vkmHSGJ/ytpJZkekNn9vaisatnjdtGSDOG77767c+dOLmsYHh4W1wNLPdOkpqbm97//vaIobW1t8T0mdmnSCUR8TGLvpdmliZ/Coh+ootsJdtGVL34YL9oGqEb6X4+3Iia+eOLEiUV3/YkTJ5KWcTgcqYulX5V4JXFV8RqSXL58OWk94gI80fnz51M3Eb9mF8ufP38+cVcsKv4ry773pLcsXk/aUEZvatl9CG3V1dWVlJScOHHi8ePHmqwwzUcmDoPEg0p8MZ+bm1sqRBOPkLlsD8js/l5UVrXscbtoSYZw6NAhi8Xyq1/96tGjR9mtIfVUsKj4B7ToPxNXJfbeUrs09SNOXY9+J9g51YexLgEWf2OLxkliBfEXU0/08d9N/cXUV1J/K/7+T5w4kbryNJtL3Ndi/eLaPHWZNAeTWD6+gMr3Hj8rxZdJ2lDWb2rRAqCtLVu2iJ1st9u//vrr169f57hCNX9i8dxK+q3UoyhxVVkfkNn9vaisSv3fvoGiS9i9e7d4I+Xl5WfPnn316lWmaxC7dNm/X7F/4kfFsgE2t9gujZ9nUs9gqcvocYKdU3fAzM3NlSm5sS491NDhcCQ+Ek3sgvPnzyc+V+bIkSOi3fbBgweJrY6Jv3vkyJETJ060tbU9ePBA/G7qqi5evDgwMCC6N4X//u//Fu8/sQbxc1tb2/nz5xNfT9zcZ599JsbkxNdfU1Nz+vTpo0ePDgwMqNknoqn6xIkT8f72jN57Glm/qdR9qIm//Mu/HBoa0mptRvf999+LH6ampk6fPt3W1tbf319aWqrrRkWrUZxonEk6QsRxeOrUqQsXLoifczwgM/17UVlV6sq1PW7/8z//8+///u9zXEl2xsbGxA8zMzNfffXV7373u/v3769cuVL9GsTJ7aOPPkq/mFgg8UyYhfh5Jv65xD+IK1euiA9C1xOs+gMm1wBblMPhOH36dNIBd/HixXg1w8PDDx48ePDgwaVLlxbd13v27En8Z11dnaIo8cwXbcGfffZZ4jK//OUvE1d16dIlRVFE826io0ePtrW1JUXRL3/5y/jP8T/dpPWrdOXKlba2tqTwzui9p5HRm0q/DzVht9vXrVun4QoN7fHjx4n/jMVi8/PzugaYw+FIShpxhKSOihZnjUuXLom//BwPyEz/XlRWJeh33FqtVlmH6/j4eNIr8/PzWaxH5TfdHC16nkk8ZpZaRtHoBKv+gMk1wOYSeoAuXLggNvn73/9+0R09PDz861//Ws0fiThq00vaRNJ3k6W+sIhYzfEbylKGh4fFJ5r0vVjJ5L2nkdGbUrMPc/Sv//qvem/CQKqrq0WGVVRUnD59+uTJk2naJ3S11EefeIRockBmRE1VaRbL3S9+8Ytf/OIXOq08vYaGhsHBQUVRbDbbb37zmy+//LK8vDyL9QwPD+chw9Rc7eXhBKvmgNFyFOLJkydFM2VdXd2iQ0rq6uri23Y4HOJWhkX7bNPIaCidyg972QtzlcQev3z5ctJ2tXrvQn6+hSFTdrtdDOIYHBw8depUHtIr6WJFUX3iyPGAzPTvJZ8xWZjsdrsYxDE4OHjmzJks0kt8NA8ePEi/mFgguxNLEjXnGZ1OsOoPGI2bEMXtCG1tbUePHp376fC8X//610pKs2YWMjp95+cLiyBuIUzqVxC0eu9CPt8U1PuP//iP+fn5PFz4Luvu3bvpjxBtD0iVlq3KxP7whz+8efNm3759Wa9BdJEs2xcoejcTm+yypuY8o+u5SM0Bo/19YPE/iaS7NESopjZrZv3tLOlSLOm7yVJfWMSloSbfUBKlDtxIpNV7z/ObQkY+/vhj6eml8nu65n+MmlRlYtXV1bmkl/JDj1H6OZ/i94pl138fp+bz0vVcpP6A0eVG5vi8JkvdXxwXXyCjg1sMzRTDYOJEv1+c+A6SeqeCeCW17SUXYuCGkhDeKn9L/KD+vefzTcGIxBHy9ddfJ70u7ndOP89Qdn+MelcFoaamRpz3lrotXXRqKopy4sSJ+IXLokNgkk6VqRY9zyTe/rzUMopG5yL1B4wuAXby5EkRoYkVxG+Nir9y4cKF1EEsaoivIWI8pXhFXAAlLiO+g7S1tcVvDhf3dYuk0XDyyvjAjTQDpTJ672kG6+ftTcGgRANAJBJJPM1duXJFHGynT58Wr2h1QGpblXq5l2REFy9eFPc51NXVXbhwIXHilQsXLohOzaTxz6LzKX4tIabDWPQ6O3GXxs8z8RNsfMLf+EGi67lI/QGj11RSYhheJBKJ7wKxVTFhiXDq1CnRe6xkOEy2pqZG/JboKrdarW1tbUn3ddfU1IjvAmKeZqvVGp/COXWQRS7iZwGxlSTiC4vK9x6/h2Opr6V5e1MwLnGEiNOcONjEn/2JEyfi3SdaHZDaVqWGhiUZUUdHhzjRnTp1Kr4n6+rqRGCcOHEiaaIpcW+foihHjx4VS0YikaTLptRdmnqCTf2w9D4XqTxgdJwLMb6jxStHjhy5fPlyvG3U4XBcvny5o6ND7L5Mv1KdPHkycW2XL19O/f4o5hRODDZx07iGd/KqpPK9x4+b9KsqkDeFwpR6hIjjLfGLuYYHpIZVqaFhSQZ18eLFpD2p/HASWHRnxjNPURSHw5F6qbDoLk06wS76Yel6LlJ5wFjm0k7lBwBAYdLrCgwAAF0RYAAAQyLAAACGRIABAAyJAAMAGBIBBgAwJAIMAGBIBBgAwJAIMACAIRFgAABDIsAAAIZEgAEADIkAAwAYEgEGADAkAgwAYEgEGADAkAgwAIAhEWAAAEMiwAAAhkSAAQAMiQADABgSAQYAMCQCDABgSAQYAMCQCDAAgCERYAAAQ/p/7B1kgUBCiMIAAAAASUVORK5CYII=)

---

# Random experiments and causality

- If the randomization is truly random, it can't possible be related to anything on the back doors. It ONLY affects Treatment - nothing else!
- So AMONG the people who were randomized in/out of treatment, this is what the diagram looks like. Easy identification!

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAIAAADOgk3lAAAABmJLR0QA/wD/AP+gvaeTAAAP4UlEQVR4nO3csYsj9cPH8VEfVk5Ywd7qFoUogp2kSrvc4VZXWh2pFrZxEVwsxOqqrQLbGA5sbDwQtpC0V+UP0LvFWyIWJ9qKirBw+Cu+z2+YJ8lmZ7PJ3n18Xq9qL5dMZibfzHsyM8lLZ2dnFQCkefl5zwAALEPAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQKTlA7Zxvl6vt7u7u8K5bGM0Gm1sbFzP8w4Gg42NjcFgsI4pNxdhfU/Ei6a81he65sEwNSBfBC/gLF2zyWSyu7vbHBW7u7uTyWTpCeau0rV8AhuPx8PhcGNjYx0T/xcbjUb7+/vPey7gf72AA/IFnKVr1uv1Op3OcDhs3jgcDjudznIRil6lVw3Y2TzHx8flf0OrfqG9vb2zs7O9vb1/zRPx3JXXuqncPnWjwfD/Wa/XG4/HVVUdHh6enJyUIXFycnJ4eFhV1XA47PV6z3ser9VaPoFtb2+fnJxUVTW1mwDAcnZ3d8fjcbfbLfsxW1tb5fatra29vb2Tk5Nutzsej/+tHxvmWtdFHPXKnTIajXq9XvNs2dQx/XJsdzQaNe859whvOelVjEajuU83dbB4djr1000mk/rp6lmqT0v0er3mU0ydmlpwxqL5dIuXvdfr7ezsVFVVjr6WUTj3HFj7hbpwHbJCT548Kftt16AM/sFgUL8Ler1e/fpOjZDmf01N5LIDcrn3S8u5unDczp2lCD/99NP3339/lSlMJpPyeeDhw4dz77C1tXX//v2qqobDYb3Gyiqd2oCUl6msvQWrtPkqzH1Bq7VtYOdOfP4wnnsMsI3FD6+PIjZv7Pf7c1d9v9+fuk+325292+JJlVuak6rnYcrx8fHUdMoH8KbDw8PZp6g/s5f7Hx4eNlfFXPVDLlz2qUUut0890aUW6sJ1yGp1Op2XX3653+8/ffp0JRNc8JKVYdAcVGXH/Ozs7LyINkfI2bIDcrn3S8u5unDczp2lCB988MFLL7300Ucf/fzzz8tNYXZTMFf9As39Z3NSZe2dt0pnX+LZ6axvA3vWehivJWD1gs3NSXMO6htnN/T1Y2cfOHvL7KPq5e/3+7MTX/B0zXVdpl8+m8/eZ8FgKvev79By2eutUn2fqSdaeqHmzgCr9eabb5aVvLm5ee/evT///POKE2zzFqu7NfWo2VHUnNTSA3K590vLuWr/3g9KV/HOO++UBblx48bnn3/++++/X3YKZZVe+P4t66ceFRcG7GzeKq23M7NbsNn7rGMDe9ZuwJydnf1PdTUb519q2O12j46OmvNa1sj29nZ94/b2djlue3p62jzq2Hzs9vZ2v98fDoenp6flsbOTOjo6evToUTm9WXz33Xdl+ZvzUP4eDoeHh4fN25tPd+vWrXJNTj39ra2tg4ODnZ2dR48etVkn5VB1v9+vz7dfatkXWHqhZtfhSnz44YdPnjxZ1dTS/fbbb+WPP/744+DgYDgc/vDDD6+88span7QcNaqVgzNTI6SMw/39/cFgUP6+4oC87Pul5VzNTny14/bbb7/99NNPrziR5fzyyy/lj7///vuLL7748ssvf/zxx1dffbX9FMrG7a233lp8t3KH5pZwCfV2pn5d6hdiNBqVF2KtG9j2A+aqAZur2+0eHBxMDbijo6N6biaTyenp6enp6YMHD+au63fffbf5z06nU1VV3fxyLPjWrVvN+9y5c6c5qQcPHlRVVQ7vNu3s7AyHw6kU3blzp/67futOTb+l0Wg0HA6n4n2pZV/gUgu1eB2uxObm5htvvLHCCUZ7+vRp85///PPPs2fP1hqwbrc7VZoyQmavii5bjQcPHpR3/hUH5GXfLy3nqljfuN3Y2Hhew/XXX3+duuXZs2dLTKflnu4Vzd3ONMfMefepVrSBbT9grhqws8YZoMFgUJ7y/v37c1f0ZDK5e/dumzdJGbWLTT3F1L7JeTssJatX3EM5z2QyKa/o1H5xdZllX+BSC9VmHV7R119/ve6nCHLz5s3SsNdff/3g4GBvb2/B8Ym1Ou+lb46QlQzIS2kzVwvudnW3b9++ffv2mia+2Pvvv//48eOqql577bVPPvnk448/vnHjxhLTmUwm19CwNp/2rmED22bArPIqxL29vXKYstPpzL2kpNPp1M/d7XbLVxnmnrNd4FKX0rV8sS/8YN5SWePHx8dTz7uqZS+uZy+My9rc3CwXcTx+/Hh/f/8a6jX1YaVqveG44oC87PvlOjP5Ytrc3CwXcTx+/Pizzz5bol7lpTk9PV18t3KH5TYsU9psZ9a0gW0/YFZ8CLF8HWE4HO7s7Jz938vz7t69W80c1lzCpTbf17PDUpSvEE6dVyhWtezFdS4U7X3zzTfPnj27hg++Fzo5OVk8QlY7IFu6cK7+xb766qu//vrrvffeW3oK5RTJhecCy9nN5iG7pbXZzqx1W9RmwKz+e2D1W2LqWxolqrOHNZfeO5v6KDa1b3LeDkv5aLiSPZSm2Qs3mla17Ne8UFzK22+//dzr1XI/feVvxpXM1b/YzZs3r1Kv6r9njBb/5lP9XbHlzt/X2rxea90WtR8wa/kic/27Jud9v7hW3+FSg7tcmlkug6mV8361sg8y+02FcsvssZerKBduVI14t3xU+aP9sl/nQpGojJB79+5N3V6+77z4d4aWezOue64otra2ynbvvK+ll5OaVVX1+/36g8vcS2CmNpWz5m5nml9/Pu8+1Yq2Re0HzFoCtre3VxLanIP6q1H1LYPBYPYiljbKbki5nrLcUj4ANe9T9kGGw2H95fDyve5SmhX+eGV94caCC6UutewLLta/toUiVDkAMB6Pm5u50WhUBtvBwUG5ZVUDcrVz1d7VZynR0dFR+Z5Dp9MZDAbNH14ZDAblpObU9c/l5FP9WaL8HMbcz9nNVVpvZ+oNbP2Dv/UgWeu2qP2AWddPSZXL8Mbjcb0KyrOWHywp9vf3y9nj6pKXyW5tbZVHlVPlGxsbw+Fw6nvdW1tbZV+g/E7zxsZG/RPOsxdZXEW9FSjPMqXssLRc9vo7HOftll7bQpGrjJCymSuDrbzt+/1+ffpkVQNytXPVxgpnKdHDhw/Lhm5/f79ek51OpwSj3+9P/dBU+W5fVVU7OzvlnuPxeOpj0+wqnd3Azr5Y694WtRwwa/wtxHpFl1u2t7ePj4/rY6Pdbvf4+Pjhw4dl9V12l2pvb685tePj49n9x/Kbws2wlS+Nr/CbvC21XPZ63Cye1AuyULyYZkdIGW/NHfMVDsgVzlUbK5ylUEdHR1NrsvrvRmDuyqybV1VVt9ud/agwd5VObWDnvlhr3Ra1HDAvnS38KT8AeDGt6xMYAKyVgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJEEDIBIAgZAJAEDIJKAARBJwACIJGAARBIwACIJGACRBAyASAIGQCQBAyCSgAEQScAAiCRgAEQSMAAiCRgAkQQMgEgCBkAkAQMgkoABEEnAAIgkYABEEjAAIgkYAJH+A32RWz71M24wAAAAAElFTkSuQmCC)

---

# Example

- What is the effect of going to the opera on your health?
- Obviously this has all sorts of endogeneity - the people who go to the opera are different in many ways that also relate to health
- Income being a big one, as well as wealth, education, being healthy enough to go to a show, etc. etc. etc.
- So many back doors there's no way we could plausibly control for everything

---

# Ahem

Of course that didn't stop it from being a real study... not a *good* study, but one that got lots of attention anyway (this is the New York Times)

---

# Opera

- But what if we could take a bunch of people and randomly assign them to go to the opera or not?
- Within that group of randomized people, we wouldn't expect any back doors
- Since within that group, "going to the opera" is *only* determined by random assignment. We've removed the Income `\(\rightarrow\)` Opera, Education `\(\rightarrow\)` Opera etc. arrows from the graph, so there are no back doors any more!
- We can easily identify the effect by comparing health among opera-goers and non-opera-goers in our randomized sample

---

# Pros and Cons of Experiments

- Experiments give us very clean identification
- Unlike controlling for stuff, we are closing back doors even for stuff we can't measure, *or don't even think of*
- And it requires fewer assumptions than things like difference-in-difference, where we're relying to some extent on assumptions about how treatment is assigned. With an experiment we *know* how treatment is assigned since we assigned it ourselves!
- (plus from a rhetorical standpoint they're very believable and easy to explain)

---

# Pros and Cons of Experiments

- On the other hand there are some difficulties with experiments that other forms of analysis don't have
- We've talked before about how experiments aren't always feasible
- They may also be less representative - if you *can* do an experiment but only on a small highly-selected group of people, your results will be *well-identified* but they won't necessarily *generalize* to the population
- That's not a trivial concern - psychology has in the past done lots of tiny experiments mainly on college students. This has led to a whole lot of results that don't replicate (because they're so small, so significant effects are often just noise), and when they do are representative of the effect *among rich-county college students* but not among everybody
- Those same problems apply to non-experimental work, it's just easier to avoid because it's easier to sample lots of different people if you don't have to do an experiment
- (although the "only studying people in rich countries" problem is way broader than just experiments)

---

# Pros and Cons of Experiments

- Another issue with experiments is that the experiment itself might change the behavior
- If people act differently because they know they're in an experiment, that's "the Hawthorne effect", named after an early social science experiment that found that people behaved differently in experimental conditions
- Further, if people have an idea of the behavior the experimenter *wants* to see, they may give it to them. This is "experimenter demand"
- Also, for a lot of experimental changes, people may adjust over time - your experiment might be great at getting people to go to the gym... but people may fall back into old habits after the data stops getting collected, even if the treatment is still in place
- In general, you have to be *very careful* that what you're getting is natural behavior, that people *understand* any questions you have to ask them, that you're really studying what you want to study
- Some of these problems can be solved with "field experiments" where people don't realize they're in an experiment and they're just behaving as normal. One example of this is when websites randomly show different versions of the site to different people and track how they respond ("A/B testing" they call it but it's just a regular ol' experiment)

---

# Pros and Cons of Experiments

- That said, despite these issues experiments are super valuable! 
- No cleaner way to get a causal effect. Hard to deny that
- We just need to be mindful of the boundaries of what experiments can give us
- As well as the procedural stuff we'll talk about here
- They're not automatically better than other methods. The problems they face are just different

---

# Concept Checks

- Even if we have an experiment that *statistically* picks up the exact effects we're looking for, what are some ways in which we might expect that it's picking up *unnatural behavior*?
- Design a hypothetical experiment. It can be about anything you like. But also, select *what you want the outcome to be*. How might you be able to tip the scales in favor of that outcome by selecting a non-representative group to do the experiment on?
- What are some things that would need to be true for us to assume that the results from an experiment apply to the whole population?

---

# Performing an Experiment

1. Figure out what needs to be randomized (the treatment, usually)
1. Figure out what you want your outcome to be
1. Figure out where you'll do the experiment and how much data you need
1. Figure out *how to randomize* the treatment
1. Perform the experiment and collect data
1. Analyze the data
1. Check for threats to the experiment

---

# Treatment and Outcome

- We're going to walk through our example as though we're going to do the opera and health study as an experiment
- Of course, we want to be sure that once we do our experiment that we've actually answered the question we want to answer
- If our question is "do the arts improve health" then using opera as a treatment doesn't really answer that
- If it's "does going to the opera improve your health" that's better
- But also... we can't just measure "health" - what is our measurement? Let's get more specific and ask "does going to the opera extend your lifespan?" which was what the actual original study was about

---

# Where and How Much?

- We need an environment where we can randomize people to attend the opera or not
- and what is "or not"? That's always a question in experiments. There's no such thing as "nothing." We could leave them to do whatever they'd *choose* to do (which might be the opera, uh-oh). We could give them an alternative treatment like "sit in a blank room and stare at a wall" (but what's the effect of that?). Do we take people who already go to opera and have them go to *one more* opera?
- Not just in experiments but in all analyses of a treatment effect we always have to ask what treatment people would be otherwise getting
- Let's compare the opera to a more common entertainment activity like the movies
- We can recruit a bunch of people and randomly assign them to either go to the opera every week (which we need a budget to pay for), or send them to the movies

---

# How Much?

- We've talked before about *statistical power* - having a large enough sample that we can have a reasonable expectation of finding a result if it's there, and just generally having small enough standard errors that the results are useful
- In experiments, though, we have some control over our sample size!
- So *before* collecting any data, we need to do a *power analysis* - what sample size do we need to get our standard errors down to a useful level? 
- A power analysis has four elements: sample size, anticipated true effect, variance of the outcome, and power. Set the values of three of them based on what you anticipate to be true, and it will tell you the fourth!
- A common goal is to have a sample large enough 80% (or 90%) chance of getting a significant result of an anticipated size

---

# How Much?

- There are a lot of calculators out there that will do these actual calculations for you, like [PowerandSampleSize](http://powerandsamplesize.com) or [G*Power](https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower.html), which is what I use
- Another great method, especially with more complex experiments, is to use simulation - randomly generate data with the properties you expect your data to have, see if you find an effect, then repeat thousands of times. Proportion of times you get a significant effect = your power. Standard deviation of the effect = standard error. Look back to our lectures on sampling variation to see some examples
- I will post a guide sheet on how to use these tools

---

# How Much?

- But the general idea of power analysis is this:
- Imagine trying to see a crater on the moon with a telescope
- The bigger your telescope (bigger sample), the more fine detail and smaller craters you can see (small effects)
- You need a *lot of people* to see a *tiny effect*
- You can answer "does a parachute save your life" with two people. To answer "does this BrainPower pill raise your IQ by .000001 points" you need about eight billion
- You also need a whole lot of people to see if the *effect differs* between different groups
- Roughly, if you need X people to see an effect overall, you need **16X** people to see whether the effect differs between two groups

---

# How Much?

- If we think that going to the opera instead of the movies will increase your lifespan by a month on average, and we are going to randomize 60-year-olds for whom the standard deviation of lifespan might be about 5 years, we are looking for an effect that is `\(1/(12*5) = 1/60\)` of a standard deviation. Pretty small!
- Any guesses how many people we'll need to recruit to have an 80% chance of finding a significant effect when we do a hypothesis test at the 95% level?

---

# How Much?

- According to the [PowerandSampleSize](http://powerandsamplesize.com) "Compare two means two-sided inequality" calculator, we need **56,397** people in our study. To only have an 80% chance it works *if* we're right about how big the effect is! FOr a 90% chance it's 75,550. 
- Ouch.
- This is because we expect the effect to be tiny. Statistical properties of tiny effects are really bad y'all. Beware!
- Maybe we can only afford 2,000 people. Then we can ask what's the "minimum detectable effect" (MDE) - i.e., how big does the effect *need to be in truth* for us to have a good chance of finding it?
- With 2,000 people, we have an 80% chance of picking up an effect of 5.5 months or bigger. So if we can only afford a sample of 2,000, we shouldn't even bother if we think the opera has a smaller effect than that (and it almost certainly has a smaller effect than that)

---

# How Much?

- By the way, all this power analysis stuff applies to observational data/non-experimental data too, it's not specific to experiments
- We just don't do it as often because we can't control the sample size anyway, and it's easier to get huge samples
- You still huge samples to reasonably study small effects
- So don't pursue effects that are likely to be really tiny, or at least tinier than your sample can handle
- If we ran the underpowered study anyway and *do* get a significant result, it would be more likely to be a false positive than a true positive. That's low power for you!

---

# Concept Checks

- How many observations, roughly, would we need to get 90% power to detect a one-unit effect when the standard deviation is 10 and we have an experiment where half the sample is randomized into treatment and control?
- Intuitively, why do we need more observations to have power to detect a small effect?
- Which of these analyses is likely to have higher statistical power: detecting the effect of a job-training program on whether you find a job in the next month, or detecting the effect of that same job-training program on the proportion of the rest of your life that you spend unemployed?

---

# How to Randomize

- The point of randomization is that the treatment and control groups are basically the same on everything, even the stuff we can't measure
- On average, this will happen naturally if we just do random assignment for everybody with a random number generator *if our sample is large enough*
- In smaller samples we may want to give it a helping hand by randomizing *within certain groups* ("blocking") to ensure those groups are evenly distributed across treatment and control
- This is especially important if we want to compare the effect within those groups

---

# How to Randomize

- Another issue is *what level will randomization be done at*?
- Often you want to randomize each individual, but often that's not possible
- For example, if you're looking at an education intervention, you might have to randomize classrooms or schools. It might not make sense to randomize individual students
- The effective sample size is somewhere between the number of students and the number of classrooms/schools. Hard to get a "big" sample when doing this!
- Power analysis gets much tougher - in these cases you generally want to do a simulation rather than use a calculator

---

# Perform the Experiment

- Be 10000% sure before you start that the data you'll get will actually answer your question - it's super expensive to REDO an experiment. Try with some simulated data first!
- Also, ensuring the experiment is performed as intended can be very tough!
- How can we be sure that our moviegoers aren't sneaking off to the opera at night?
- Or that people show up to the opera you've assigned them to?
- Or that the team you've employed to hand out the opera and movie tickets don't mix them up?
- Or that opera-goers won't sneak their movie-goer friends into the opera?
- Or that we can actually get the lifespan data we want? Who do we get that data from? Can we trust it?

---

# Analyze the Data

- Once we have the data, it's pretty darn easy to analyze! That's the beauty of experiments
- Just compare the treated and untreated groups. Done! No back doors to close, ideally

```r
lm(outcome ~ treatment, data = experimentdata)
```

- Treatment is binary (or categorical if we have multiple treatments), we know how to interpret that!

---

# Threats to the Experiment

So far so good. So if we do all this, *what can go wrong* and *how can we check*?

Some things that can go wrong:

- Attrition
- Balance
- Compliance

---

# Attrition

- If your experiment has *any* gap of time between randomization into treatment and collecting the outcome, you will likely have some *attrition* from your study
- People will become uncontactable, or stop reporting data, or just generally *leave*
- This can be a problem if it happens for non-random reasons, because that reintroduced endogeneity!
- Say people don't like opera, and so people assigned to see opera are more likely to stop showing up to the experiment than people assigned to the movies. Or perhaps people with more sensitive hearing (and worse health in other ways too on average) are more likely to drop out of the opera group than the movie group
- Now your treated group represents people who were randomly assigned to treatment *and didn't dislike opera enough to leave*. That could be related to the outcome!

---

# Attrition

- What can we do about it?
- Keep in mind, what we're really concerned about is *whether attrition is different in the treated or untreated groups*
- Either *higher/lower* in treated/untreated, or *among different groups of people* in treated/untreated
- First we can *check* whether any attrition we had seems to be non-random

---

# Detecting Attrition

Create a binary variable telling us there's attrition, which we might see by not having an outcome for that individual

```r
experimentdata <- experimentdata %>%
  mutate(attrition = is.na(outcome))
```

Check whether it's higher/lower in the treated/untreated group:

```r
lm(attrition ~ treatment, data = experimentdata)
```

Check whether it's different in treated/untreated *for a specific group* - this could be some pre-experiment demographic group like gender, or could be some early outcome, like "people who rated their experience lower after the first week of the experiment"

```r
lm(attrition ~ treatment*gender, data = experimentdata)
```

---

# Dealing with Attrition

If we find evidence of attrition, what should we do?

- Trim the data: if, for example, we think those with the weakest outcomes are the ones attriting more in the treatment group, we could drop the actual weakest outcomes in the control group until attrition rates are equal
- Generally, trim in the direction that makes it harder to support your hypothesis
- Or, we can use a **selection model** using the "Heckman correction" which is beyond the scope of this class. But basically you treat attrition as a form of omitted variable bias, predict attrition using your controls, and use that to adjust for attrition

---

# Balance

- Sometimes, just through bad luck (and smaples smaller than infinity), randomization doesn't perfectly control for variables on the back door
- Is this a problem? Certainly can be. If it implies randomization was done improperly, that's bad. Also if you now have treatment/control differences related to variables on back doors, that's bad
- We can check with a *balance table* that compares means of control variables across treated/untreated and tests if they're different

```r
library(vtable)
sumtable(experimentdata, group = 'treatment', group.test = TRUE)
```

- If we see more differences than we expect (say, a lot more than 1/20 comparisons are different using a 95% test) that may indicate problems with randomization
- If we see any differences, we may want to add them as controls to the analysis if the difference is *meaningfully large* not just statistically significant

---

# Balance Tables

```r
data(mtcars)
sumtable(mtcars, group = 'am', group.test = TRUE)
```

<table>
<caption>Summary Statistics</caption>
 <thead>
<tr>
<th style="border-bottom:hidden; padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="1"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">am</div></th>
<th style="border-bottom:hidden; padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="3"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">0</div></th>
<th style="border-bottom:hidden; padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="3"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">1</div></th>
<th style="border-bottom:hidden" colspan="1"></th>
</tr>
  <tr>
   <th style="text-align:left;"> Variable </th>
   <th style="text-align:left;"> N </th>
   <th style="text-align:left;"> Mean </th>
   <th style="text-align:left;"> SD </th>
   <th style="text-align:left;"> N </th>
   <th style="text-align:left;"> Mean </th>
   <th style="text-align:left;"> SD </th>
   <th style="text-align:left;"> Test </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> mpg </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 17.147 </td>
   <td style="text-align:left;"> 3.834 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 24.392 </td>
   <td style="text-align:left;"> 6.167 </td>
   <td style="text-align:left;"> F=16.86<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> cyl </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 6.947 </td>
   <td style="text-align:left;"> 1.545 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 5.077 </td>
   <td style="text-align:left;"> 1.553 </td>
   <td style="text-align:left;"> F=11.272<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> disp </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 290.379 </td>
   <td style="text-align:left;"> 110.172 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 143.531 </td>
   <td style="text-align:left;"> 87.204 </td>
   <td style="text-align:left;"> F=16.122<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> hp </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 160.263 </td>
   <td style="text-align:left;"> 53.908 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 126.846 </td>
   <td style="text-align:left;"> 84.062 </td>
   <td style="text-align:left;"> F=1.886<sup></sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> drat </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 3.286 </td>
   <td style="text-align:left;"> 0.392 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 4.05 </td>
   <td style="text-align:left;"> 0.364 </td>
   <td style="text-align:left;"> F=30.97<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> wt </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 3.769 </td>
   <td style="text-align:left;"> 0.777 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 2.411 </td>
   <td style="text-align:left;"> 0.617 </td>
   <td style="text-align:left;"> F=27.642<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> qsec </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 18.183 </td>
   <td style="text-align:left;"> 1.751 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 17.36 </td>
   <td style="text-align:left;"> 1.792 </td>
   <td style="text-align:left;"> F=1.674<sup></sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> vs </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 0.368 </td>
   <td style="text-align:left;"> 0.496 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 0.538 </td>
   <td style="text-align:left;"> 0.519 </td>
   <td style="text-align:left;"> F=0.875<sup></sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> gear </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 3.211 </td>
   <td style="text-align:left;"> 0.419 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 4.385 </td>
   <td style="text-align:left;"> 0.506 </td>
   <td style="text-align:left;"> F=51.197<sup>***</sup> </td>
  </tr>
  <tr>
   <td style="text-align:left;"> carb </td>
   <td style="text-align:left;"> 19 </td>
   <td style="text-align:left;"> 2.737 </td>
   <td style="text-align:left;"> 1.147 </td>
   <td style="text-align:left;"> 13 </td>
   <td style="text-align:left;"> 2.923 </td>
   <td style="text-align:left;"> 2.178 </td>
   <td style="text-align:left;"> F=0.1<sup></sup> </td>
  </tr>
</tbody>
<tfoot>
<tr>
<td style = 'padding: 0; border:0;' colspan='100%'><sup></sup> Statistical significance markers: * p&lt;0.1; ** p&lt;0.05; *** p&lt;0.01</td>
</tr>
</tfoot>
</table>

---

# Balance Tables

- If that were an actual experimemnt, we'd be concerned!
- The F-test that treatment is unrelated to the background rejects the null of no difference for `mpg`, `cyl`, `disp`, `drat`, `wt`, and `gear`!  Plus, all of these differences look pretty big in absolute term
- Way more than we'd expect by chance
- (the only one that it's totally okay if it's different is the outcome variable)
- We'd (a) be concerned that randomization didn't work well here (which, of course it didn't, this isn't an actual experiment), and (b) want to add these variables as controls
- If it's just a stray one or two, adding a control is probably good. If we see LOTS of significant differences, then the randomization might have problems and we're back to worrying about back doors through variables we DIDN'T test

---

# Compliance

- When you run an experiment, you can certainly *assign* people to treatment but you can't *make them do it*
- So what happens if they don't? You might get some movie-goers sneaking out to the opera, and some opera-goers skipping it entirely, or heading to the movies instead
- That means we have *non-compliance*
- Especially a problem if it's non-random, since that brings endogeneity back in
- What can we do about that?

---

# Intention-to-treat

- If we have some noncompliance and just ignore the problem, we end up with an *intent-to-treat* (ITT) result
- Basically, it all still works, except the effect we get isn't *the effect of treatment*, it's *the effect of being assigned to treatment*, which is different
- This can still be handy, especially if treatment might be assigned the same way in the future!
- This will in general *underestimate* the effect of the treatment itself, since we include people in the "treated" category who weren't actually treated, and people in the "untreated" category who were, so the two groups get closer together. Smaller effect!

---

# Two stage least squares

- a.k.a. instrumental variables
- Or, if we can observe whether people actually received treatment (separate from us assigning it to them), we can use two-stage least squares (2SLS) to adjust the ITT so that we get the effect of actual treatment instead
- Basically, 2SLS takes the effect of assignment and scales it up by how much the treatment assignment increases the treatment rate
- If being assigned to opera increases the chances you actually go to the opera from 20% to 80%, and we see that the assigned-to-opera group lives 3 months longer, we got that 3-month effect despite only 60% additional people seeing the opera! So the effect of opera if EVERYONE had done what we said is (3 months)/(60%) = 5 months.
- Keeping in mind this is more representative of the effect *among those who respond really strongly to treatment*
- And doesn't work if people *intentionally do the opposite of what you say*
- (See the instrumental variables module for more information)

---

# Two stage least squares

- If it's more complex - you have control variables, etc., you can't just do the scaling, and actually have to perform two-stage least squares.
- Remember `lm_robust()` in `estimatr`? We conveniently also have `iv_robust()`
- If `treatment` is "actually getting treatment" and `assignment` is "being assigned to treatment" then we have

```r
library(estimatr)
iv_robust(outcome ~ treatment | assignment, data = experimentdata)
```

---

# Two stage least squares

- Does it actually work? Let's see!

```r
tb <- tibble(assignment = sample(c(TRUE,FALSE),1000, replace = TRUE)) %>%
  # Start with assignment working
  mutate(treatment = assignment) %>%
  # Randomly reassign 20% of people not based on treatment at all
  mutate(treatment = case_when(
    sample(c(TRUE,FALSE),1000,prob = c(.8,.2), replace = TRUE) ~ assignment,
    TRUE ~ sample(c(TRUE,FALSE),1000, replace = TRUE))) %>%
  # True effect is 2
  mutate(outcome = 2*treatment + 3*rnorm(1000))
# Intent-to-treat
itt <- lm_robust(outcome ~ treatment, data = tb)
# 2sls to adjust for compliance
twosls <- iv_robust(outcome ~ treatment | assignment, data = tb)
# Do the scaling by hand (since this is a simple case)
treatment_increase <- (tb %>% filter(assignment) %>% pull(treatment) %>% mean()) - (tb %>% filter(!assignment) %>% pull(treatment) %>% mean())
assignment_effect <- (tb %>% filter(assignment) %>% pull(outcome) %>% mean()) - (tb %>% filter(!assignment) %>% pull(outcome) %>% mean())
scaled <- assignment_effect/treatment_increase
```

---

# Two stage least squares

```r
c(assignment_effect, treatment_increase, scaled)
```

```
## [1] 1.5066971 0.7859563 1.9170240
```

```r
export_summs(itt, twosls, statistics = c(N = 'nobs'))
```

<table class="huxtable" style="border-collapse: collapse; border: 0px; margin-bottom: 2em; margin-top: 2em; ; margin-left: auto; margin-right: auto;  " id="tab:unnamed-chunk-14">
<col><col><col><tr>
<th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 1</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 2</th></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(Intercept)</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">-0.07    </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">-0.14    </td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.13)   </td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.14)   </td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">treatmentTRUE</th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">1.78 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">1.92 ***</td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.18)   </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.23)   </td></tr>
<tr>
<th style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">N</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">1000       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0.8pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;">1000       </td></tr>
<tr>
<th colspan="3" style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt;    padding: 6pt 6pt 6pt 6pt; font-weight: normal;"> *** p < 0.001;  ** p < 0.01;  * p < 0.05.</th></tr>
</table>

---

# Concept Checks

- Why do attrition and compliance only cause *really bad* problems when they're non-random?
- How do we know that random non-compliance *shrinks* the effect, as opposed to estimating it to be too big?
- Say we have 50 covariates, and we create a balance table for them. We get 3 statistically significant differences. Should we be concerned about our randomization procedure? Should we add controls to our analysis?
- Say we are doing an experiment where we have no covariates. We get some attrition. What should we do?
- Say we are doing an experiment where we don't have any information on whether someone was actually treated, just whether we assigned them to treatment. What should we be concerned about?

---

# Next!

To complete the module (due by the end of the second-to-last week of class):

- Do the Experiments Swirl
- Do the Experiments homework
- Read the assigned paper and answer the questions